REISER4: fix for reiser4_write_extent

REISER4: fix for reiser4_write_extent

[Ignatich]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

While trying to find the cause of problems with reiser4 in recent 
kernels I came across this.

Incomplete write handling seem to be missing from reiser4_write_extent() 
thanks to reiser4-temp-fix.patch. Strangely, there is a patch by Edward 
Shishkin that should address that issue, but it is missing from -mm 
tree. Please check.

    Max


[-- Attachment #2: reiser4-fix-write_extent.patch --]
[-- Type: text/plain, Size: 6591 bytes --]

------------------------------------------------------
Subject: reiser4: fix write_extent
From: Edward Shishkin <[EMAIL PROTECTED]>

. Fix reiser4_write_extent():
   1) handling incomplete writes missed in reiser4-temp-fix.patch
   2) bugs in the case of returned errors


Signed-off-by: Andrew Morton <[EMAIL PROTECTED]>
---

 fs/reiser4/plugin/item/extent_file_ops.c |   64 ++++++++++++---------
 1 file changed, 37 insertions(+), 27 deletions(-)

diff -puN fs/reiser4/plugin/item/extent_file_ops.c~reiser4-fix-write_extent 
fs/reiser4/plugin/item/extent_file_ops.c
--- a/fs/reiser4/plugin/item/extent_file_ops.c~reiser4-fix-write_extent
+++ a/fs/reiser4/plugin/item/extent_file_ops.c
@@ -941,15 +941,15 @@ static int write_extent_reserve_space(st
  * reiser4_write_extent - write method of extent item plugin
  * @file: file to write to
  * @buf: address of user-space buffer
- * @write_amount: number of bytes to write
- * @off: position in file to write to
+ * @count: number of bytes to write
+ * @pos: position in file to write to
  *
  */
 ssize_t reiser4_write_extent(struct file *file, const char __user *buf,
                             size_t count, loff_t *pos)
 {
        int have_to_update_extent;
-       int nr_pages;
+       int nr_pages, nr_dirty;
        struct page *page;
        jnode *jnodes[WRITE_GRANULARITY + 1];
        struct inode *inode;
@@ -958,7 +958,7 @@ ssize_t reiser4_write_extent(struct file
        int i;
        int to_page, page_off;
        size_t left, written;
-       int result;
+       int result = 0;
 
        inode = file->f_dentry->d_inode;
        if (write_extent_reserve_space(inode))
@@ -972,10 +972,12 @@ ssize_t reiser4_write_extent(struct file
 
        BUG_ON(get_current_context()->trans->atom != NULL);
 
+       left = count;
        index = *pos >> PAGE_CACHE_SHIFT;
        /* calculate number of pages which are to be written */
        end = ((*pos + count - 1) >> PAGE_CACHE_SHIFT);
        nr_pages = end - index + 1;
+       nr_dirty = 0;
        assert("", nr_pages <= WRITE_GRANULARITY + 1);
 
        /* get pages and jnodes */
@@ -983,22 +985,17 @@ ssize_t reiser4_write_extent(struct file
                page = find_or_create_page(inode->i_mapping, index + i,
                                           reiser4_ctx_gfp_mask_get());
                if (page == NULL) {
-                       while(i --) {
-                               unlock_page(jnode_page(jnodes[i]));
-                               page_cache_release(jnode_page(jnodes[i]));
-                       }
-                       return RETERR(-ENOMEM);
+                       nr_pages = i;
+                       result = RETERR(-ENOMEM);
+                       goto out;
                }
-
                jnodes[i] = jnode_of_page(page);
                if (IS_ERR(jnodes[i])) {
                        unlock_page(page);
                        page_cache_release(page);
-                       while (i --) {
-                               jput(jnodes[i]);
-                               page_cache_release(jnode_page(jnodes[i]));
-                       }
-                       return RETERR(-ENOMEM);
+                       nr_pages = i;
+                       result = RETERR(-ENOMEM);
+                       goto out;
                }
                /* prevent jnode and page from disconnecting */
                JF_SET(jnodes[i], JNODE_WRITE_PREPARED);
@@ -1009,7 +1006,6 @@ ssize_t reiser4_write_extent(struct file
 
        have_to_update_extent = 0;
 
-       left = count;
        page_off = (*pos & (PAGE_CACHE_SIZE - 1));
        for (i = 0; i < nr_pages; i ++) {
                to_page = PAGE_CACHE_SIZE - page_off;
@@ -1050,14 +1046,26 @@ ssize_t reiser4_write_extent(struct file
                        flush_dcache_page(page);
                        kunmap_atomic(kaddr, KM_USER0);
                }
-
-               written = filemap_copy_from_user(page, page_off, buf, to_page);
+               written = filemap_copy_from_user_atomic(page, page_off, buf,
+                                                       to_page);
+               if (written != to_page)
+                       /* Do it the slow way */
+                       written = filemap_copy_from_user_nonatomic(page,
+                                                                  page_off,
+                                                                  buf,
+                                                                  to_page);
+               if (unlikely(written != to_page)) {
+                       unlock_page(page);
+                       result = RETERR(-EFAULT);
+                       break;
+               }
                flush_dcache_page(page);
                reiser4_set_page_dirty_internal(page);
                unlock_page(page);
+               nr_dirty ++;
+
                mark_page_accessed(page);
                SetPageUptodate(page);
-               page_cache_release(page);
 
                if (jnodes[i]->blocknr == 0)
                        have_to_update_extent ++;
@@ -1067,27 +1075,29 @@ ssize_t reiser4_write_extent(struct file
                left -= to_page;
                BUG_ON(get_current_context()->trans->atom != NULL);
        }
-
        if (have_to_update_extent) {
-               update_extents(file, jnodes, nr_pages, *pos);
+               update_extents(file, jnodes, nr_dirty, *pos);
        } else {
-               for (i = 0; i < nr_pages; i ++) {
+               for (i = 0; i < nr_dirty; i ++) {
+                       int ret;
                        spin_lock_jnode(jnodes[i]);
-                       result = reiser4_try_capture(jnodes[i],
+                       ret = reiser4_try_capture(jnodes[i],
                                                     ZNODE_WRITE_LOCK, 0);
-                       BUG_ON(result != 0);
+                       BUG_ON(ret != 0);
                        jnode_make_dirty_locked(jnodes[i]);
                        spin_unlock_jnode(jnodes[i]);
                }
        }
-
+ out:
        for (i = 0; i < nr_pages; i ++) {
+               page_cache_release(jnode_page(jnodes[i]));
                JF_CLR(jnodes[i], JNODE_WRITE_PREPARED);
                jput(jnodes[i]);
        }
+       /* the only errors handled so far is ENOMEM and
+          EFAULT on copy_from_user  */
 
-       /* the only error handled so far is EFAULT on copy_from_user  */
-       return (count - left) ? (count - left) : -EFAULT;
+       return (count - left) ? (count - left) : result;
 }
 
 static inline void zero_page(struct page *page)

Re: Reiser4. BEST FILESYSTEM EVER? I need help.

[johnrobertbanks]

Hi Ignatich,

After seeing the following benchmarks at 

http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

The Reiser4 benchmarks are so good, I have decided to try the Reiser4
filesystem.

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.

Column one measures the time taken to complete the bonnie++ benchmarking
test (run with the parameters bonnie++ -n128:128k:0)

Column two, Disk Usage: measures the amount of disk used to store 655MB
of raw data (which was 3 different copies of the Linux kernel sources).

Anyway, I have patched the 2.6.20 kernel and have a partition formatted
with Reiser4.

However, I am having trouble getting LILO or GRUB working (with
Reiser4).

Could you guys who know all about this, help me, or point me to some
help.

Thanks a lot, John.


On Fri, 06 Apr 2007 02:42:35 +0400, "Ignatich" <ignatich@gmail.com>
said:
> While trying to find the cause of problems with reiser4 in recent 
> kernels I came across this.
> 
> Incomplete write handling seem to be missing from reiser4_write_extent() 
> thanks to reiser4-temp-fix.patch. Strangely, there is a patch by Edward 
> Shishkin that should address that issue, but it is missing from -mm 
> tree. Please check.
> 
>     Max
> 
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - And now for something completely different…

Re: Reiser4. BEST FILESYSTEM EVER? I need help.

[H. Peter Anvin]

johnrobertbanks@fastmail.fm wrote:
> 
> Anyway, I have patched the 2.6.20 kernel and have a partition formatted
> with Reiser4.
> 
> However, I am having trouble getting LILO or GRUB working (with
> Reiser4).
> 
> Could you guys who know all about this, help me, or point me to some
> help.
> 

Make your /boot a separate partition and format it as conservatively as 
possible (e.g. ext3, or even ext2.)

Problem solved.

	-hpa

Re: Reiser4. BEST FILESYSTEM EVER? I need help.

[johnrobertbanks]

Yeap, I guess that will probably work. 

And here I was trying to compile old versions of GRUB from namesys.com.

By the way, do you think the benchmarks from:

http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

are accurate?

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.


Column one measures the time taken to complete the bonnie++ benchmarking
test (run with the parameters bonnie++ -n128:128k:0)

Column two, Disk Usage: measures the amount of disk used to store 655MB
of raw data (which was 3 different copies of the Linux kernel sources).

Thanks for that, John.


On Thu, 05 Apr 2007 17:23:23 -0700, "H. Peter Anvin" <hpa@zytor.com>
said:
> johnrobertbanks@fastmail.fm wrote:
> > 
> > Anyway, I have patched the 2.6.20 kernel and have a partition formatted
> > with Reiser4.
> > 
> > However, I am having trouble getting LILO or GRUB working (with
> > Reiser4).
> > 
> > Could you guys who know all about this, help me, or point me to some
> > help.
> > 
> 
> Make your /boot a separate partition and format it as conservatively as 
> possible (e.g. ext3, or even ext2.)
> 
> Problem solved.
> 
> 	-hpa
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Send your email first class

Re: Reiser4. BEST FILESYSTEM EVER? I need help.

[H. Peter Anvin]

johnrobertbanks@fastmail.fm wrote:
> Yeap, I guess that will probably work. 
> 
> And here I was trying to compile old versions of GRUB from namesys.com.
> 
> By the way, do you think the benchmarks from:
> 
> http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> are accurate?
> 

Accurate, probably.  Whether or not they're *relevant* is a totally 
different ball of wax.

	-hpa

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Hi Peter,

You say that the results may be accurate, but not relevant.

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.

If they are accurate,.... THEN they are obviously very relevant.

Trying to follow http://linuxhelp.150m.com/resources/fs-benchmarks.htm 

I have set up a Reiser4 partition with gzip compression, here is the
difference in disk usage of a typical Debian installation on two 10GB
partitions, one with Reiser3 and the other with Reiser4.

debian:/# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             10490104   6379164   4110940  61% /3
/dev/sda7              9967960   2632488   7335472  27% /7

Partitions 3 and 7 have exactly the same data on them (the typical
Debian install).

The partitions are exactly the same size (although df records different
sizes).

Partition 3 is Reiser3 -- uses 6.4 GB.
Partition 7 is Reiser4 -- uses 2.6 GB.

So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
info).

This seems very relevant to me.

John.



On Thu, 05 Apr 2007 17:39:58 -0700, "H. Peter Anvin" <hpa@zytor.com>
said:
> johnrobertbanks@fastmail.fm wrote:
> > Yeap, I guess that will probably work. 
> > 
> > And here I was trying to compile old versions of GRUB from namesys.com.
> > 
> > By the way, do you think the benchmarks from:
> > 
> > http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> > http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> > 
> > are accurate?
> > 
> 
> Accurate, probably.  Whether or not they're *relevant* is a totally 
> different ball of wax.
> 
> 	-hpa
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - mmm... Fastmail...

Re: Reiser4. BEST FILESYSTEM EVER.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

On Thu, 05 Apr 2007 18:34:48 PDT, johnrobertbanks@fastmail.fm said:

> If they are accurate,.... THEN they are obviously very relevant.

Erm. No. They're not "obviously" very relevant.

I could hypothetically create a benchmark, that's accurate and repeatable,
that shows that reiser4 is able to wash a herd of elephants exactly 11.458%
faster than ext3.  And you would, of course, say "But elephants have nothing to
do with file systems", Because they aren't relevant to file systems.

Similarly, we've seen benchmarks that show some patch improves NUMA performance
by 5% - and those aren't relevant on my laptop because my laptop doesn't do
NUMA.  And a benchmark of file system performance is only as relevant as it
reflects *your* application's use of the filesystem - how fast it can create
and remove tiny files isn't relevant if your use of the filesystem is to store
large files with long sequential read/write patterns.  And the level of
compression isn't very relevant if you're using the partition to store
already-compressed audio or video.

I know somebody who defines a "relevance index" for things, and the measure
is "how many cubicles do I have to go to find somebody who actually cares
about ABC?" - and for him, that's itself a relevant index, because if it's
0, *he* cares, and if it's 1, his immediate neighbors care and will cause him
grief if ABC is a problem.   People who are 5 or 6 cubicles away are less
likely to give him a hard time, and the people who are 15 to 20 cubicles away
are in an entirely separate building. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Reiser4. BEST FILESYSTEM EVER.

[H. Peter Anvin]

johnrobertbanks@fastmail.fm wrote:
> Hi Peter,
> 
> You say that the results may be accurate, but not relevant.
> 

NO, I said that whether they're accurate is another matter.

> If they are accurate,.... THEN they are obviously very relevant.

Crap-o-la.  Whether or not they're relevant depends on how well they 
happen to reflect your particular usage pattern.

There are NO benchmarks which are relevant to all users.  Understanding 
whether or not a benchmark is relevant to one's particular application 
is one of the trickiest things about benchmarks.

	-hpa

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Hi Peter,

You say that the results may be accurate, but "Whether or not they're
*relevant* is a totally different ball of wax." and

"Whether or not they're relevant depends on how well they happen to
reflect your particular usage pattern."

Well, surprise, surprise,.. everyone knows that.

Have a look at the (summary) of the results: 

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.


for the full results see:
http://linuxhelp.150m.com/resources/fs-benchmarks.htm 

Don't you agree, that "If they are accurate,.... THEN they are obviously
very relevant."

I have set up a Reiser4 partition with gzip compression, here is the
difference in disk usage of a typical Debian installation on two 10GB
partitions, one with Reiser3 and the other with Reiser4.

debian:/# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             10490104   6379164   4110940  61% /3
/dev/sda7              9967960   2632488   7335472  27% /7

Partitions 3 and 7 have exactly the same data on them (the typical
Debian install).

The partitions are exactly the same size (although df records different
sizes).

Partition 3 is Reiser3 -- uses 6.4 GB.
Partition 7 is Reiser4 -- uses 2.6 GB.

So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
info).

Don't you think this result is significant in itself?

Following your hint I have booted /dev/sda7 and all the programs seem to
work fine. They do not seem to be any faster than when using Reiser3.

The whole system seems about as responsive as always.

For fun, I ran bonnie++. Here are the results:

debian:/# ./bonnie++ -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93c       ------Sequential Output------ --Sequential Input-
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
debian           1G   121  99 86524  21 63297  41   920  99 187762  80 
1782 233
Latency             82484us     386ms     438ms   26758us     110ms    
398ms
Version 1.93c       ------Sequential Create------ --------Random
Create--------
debian              -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
              /sec %CP
                 16 +++++ +++ +++++ +++ 18509  92 17776  86 +++++ +++
                 19495  91
Latency               210us    5475us    5525us    5777us    5522us   
5839us

I particularly liked the 233%CP for Random-Seeks.

John.



On Thu, 05 Apr 2007 21:07:28 -0700, "H. Peter Anvin" <hpa@zytor.com>
said:
> johnrobertbanks@fastmail.fm wrote:
> > Hi Peter,
> > 
> > You say that the results may be accurate, but not relevant.
> > 
> 
> NO, I said that whether they're accurate is another matter.
> 
> > If they are accurate,.... THEN they are obviously very relevant.
> 
> Crap-o-la.  Whether or not they're relevant depends on how well they 
> happen to reflect your particular usage pattern.
> 
> There are NO benchmarks which are relevant to all users.  Understanding 
> whether or not a benchmark is relevant to one's particular application 
> is one of the trickiest things about benchmarks.
> 
> 	-hpa
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Email service worth paying for. Try it for free

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

On Fri, 6 Apr 2007 11:21:19 -0400, "Jan Harkes" <jaharkes@cs.cmu.edu>
said:
> Do you really have to repeat the results in every email you sent?

The following benchmarks are from

http://linuxhelp.150m.com/resources/fs-benchmarks.htm or,
http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.


Column one measures the time taken to complete the bonnie++ benchmarking
test (run with the parameters bonnie++ -n128:128k:0)

Column two, Disk Usage: measures the amount of disk used to store 655MB
of raw data (which was 3 different copies of the Linux kernel sources).

> Do you really have to repeat the results in every email you sent?

Damn, I did it again. WHY DO YOU CARE?

Look, its simple, I am (among other things) discussing these results, so
people need to see them.

> > Don't you agree, that "If they are accurate,.... THEN they are obviously
> > very relevant."
> 
> Not everyone does. I care mostly about reliability and availability
> neither of which are shown by your results.

Actually, to some extent, bonnie++ tests the reliability of the
filesystem, eg, NTFS-3g usually fails.

By the way, I have pulled the plug on my REISER4 system, a number of
times now, and it recovers without problem.

> With compression there is a pretty high probability that one corrupted
> byte or disk block will result in loss of a considerably larger amount
> of data. 

Bad blocks are NOT dealt with by the filesystem,... so your comment is
irrelevant, or just plain wrong.

If your filesystem is writing to bad blocks, then throw away your
operating system.

> > I have set up a Reiser4 partition with gzip compression, here is the
> > difference in disk usage of a typical Debian installation on two 10GB
> > partitions, one with Reiser3 and the other with Reiser4.
> > 
> > debian:/# df
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > /dev/sda3             10490104   6379164   4110940  61% /3
> > /dev/sda7              9967960   2632488   7335472  27% /7
> ...
> > Partition 3 is Reiser3 -- uses 6.4 GB.
> > Partition 7 is Reiser4 -- uses 2.6 GB.
> > 
> > So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
> > 6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
> > info).
> 
> Wow, consider me totally and completely, unimpressed.
> 

Wow, consider me totally impressed by your AMAZING BIAS.

Would you like to tell me why you are SO BIASED against REISER4.

John.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - A fast, anti-spam email service.

Re: Reiser4. BEST FILESYSTEM EVER.

[Jan Harkes]

Since you decide to publically respond to a private email, but not only
you did not 'discuss' anything I wrote and in fact cut out most of the
useful information in my reply I guess I will have to repeat my
observations.

Once I send out this email, I'll just add you to my friendly killfile
(as well as this thread's subject/msgids) so don't bother continuing you
one-sided 'discussions' on this topic.

On Fri, Apr 06, 2007 at 07:47:36PM -0700, johnrobertbanks@fastmail.fm wrote:
> > Do you really have to repeat the results in every email you sent?
> 
> Damn, I did it again. WHY DO YOU CARE?

Because I saw them the first time around. And although the performance
difference on those micro benchmarks seems quite impressive, I'm not
convinced.

> Look, its simple, I am (among other things) discussing these results, so
> people need to see them.

However, you do not discuss, you just repeat, and repeat, and repeat.
But for what reason. Do you want an actual discussion, or do you hate
the reiserfs developers so much that you want to antagonize any and all
other Linux file systems developers?

> > > Don't you agree, that "If they are accurate,.... THEN they are obviously
> > > very relevant."
> > 
> > Not everyone does. I care mostly about reliability and availability
> > neither of which are shown by your results.
> 
> Actually, to some extent, bonnie++ tests the reliability of the
> filesystem, eg, NTFS-3g usually fails.
> 
> By the way, I have pulled the plug on my REISER4 system, a number of
> times now, and it recovers without problem.

Very nice for you. I guess you have also not been hit by out of memory
conditions or failing partial writes. So I guess we can ignore the patch
that was just sent a day or two ago to the mailing list because you
succesfully pulled the plug, a number of times at that.

> > > I have set up a Reiser4 partition with gzip compression, here is the
> > > difference in disk usage of a typical Debian installation on two 10GB
> > > partitions, one with Reiser3 and the other with Reiser4.
> > > 
> > > debian:/# df
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > /dev/sda3             10490104   6379164   4110940  61% /3
> > > /dev/sda7              9967960   2632488   7335472  27% /7
> > ...
> > > Partition 3 is Reiser3 -- uses 6.4 GB.
> > > Partition 7 is Reiser4 -- uses 2.6 GB.
> > > 
> > > So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
> > > 6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
> > > info).
> > 
> > Wow, consider me totally and completely, unimpressed.
> > 

Here is the part of my email that you seemed to totally ignore,

    You've just saved yourself $3.80, now go get yourself a latte.
    (see. http://tomayko.com/weblog/2006/09/11/that-dilbert-cartoon)

    Seriously, disk storage is getting less expensive all the time, you can
    already buy a 250GB SATA drive for $70. Also, compression doesn't help
    once you store already compressed data such as jpeg images, mp3 files,
    or mpeg2/4/divx video files. Not only are the savings non-existant, but
    we still end up with the disadvantage that corruption propagates to
    multiple files because of the compression in the file system.

    And if it doesn't propagate across multiple files, the compression can't
    be all that good since it can't benefit as much from similarity between
    files. So if that is the case and you really want to save diskspace you
    almost have to look at read-only compressed filesystems such as cramfs,
    squashfs, zisofs, cloop and various other variants in combination with
    a unionfs overlay to get read/write functionality.

    But in the end everything is a tradeoff. You can save diskspace, but
    increase the cost of corruption. Use a better compression algorithm, but
    that uses more CPU or which is visible in performance of the
    application. This can be offset by caching more, and being lazier with
    writebacks, but that hurts on-disk consistency, creates more memory
    pressure (more swapout/paging) and generally slows down other
    applications that aren't actually accessing the disk. Having a fast
    multi-core cpu and lots of memory helps a lot, but at some point what
    tradeoff did we just make, we saved a couple of dollars not having to
    buy a larger disk, but we spend considerably more on the more expensive
    cpu and memory.

> Wow, consider me totally impressed by your AMAZING BIAS.
> 
> Would you like to tell me why you are SO BIASED against REISER4.

And that is the reponse I get, I thought you wanted discussion, but
clearly you don't care about any meaningful discussion. Your goal seems
to be to make sure that other developers end up ignoring any thread that
has reiser in the subject. And even if they are not biased and welcome a
discussion, you will just call them out on it, because clearly if they
want to discuss something they aren't totally with it, so they have to
be totally BIASED against REISER4.

At least it looks like we agree on something, I think.

Jan

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

On Fri, 6 Apr 2007 23:30:49 -0400, "Jan Harkes" <jaharkes@cs.cmu.edu>
said:
> 
> Since you decide to publically respond to a private email, but not only
> you did not 'discuss' anything I wrote and in fact cut out most of the
> useful information in my reply I guess I will have to repeat my
> observations.

You are a funny guy Jan.

Here you are, once again, cutting out my most useful information, ie,
the data I was discussing, while complaining that I cut out your most
useful information. 

You know,... you cut out this bit:

-----------------------------------------------------------------

> The following benchmarks are from
> 
> http://linuxhelp.150m.com/resources/fs-benchmarks.htm or,
> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> .-------------------------.
> | FILESYSTEM | TIME |DISK |
> | TYPE       |(secs)|USAGE|
> .-------------------------.
> |REISER4 lzo | 1938 | 278 |
> |REISER4 gzip| 2295 | 213 |
> |REISER4     | 3462 | 692 |
> |EXT2        | 4092 | 816 |
> |JFS         | 4225 | 806 |
> |EXT4        | 4408 | 816 |
> |EXT3        | 4421 | 816 |
> |XFS         | 4625 | 779 |
> |REISER3     | 6178 | 793 |
> |FAT32       |12342 | 988 |
> |NTFS-3g     |10414 | 772 |
> .-------------------------.
> 
> Column one measures the time taken to complete the bonnie++ benchmarking test (run with the parameters bonnie++ -n128:128k:0)
> 
> Column two, Disk Usage: measures the amount of disk used to store 655MB of raw data (which was 3 different copies of the Linux kernel sources).

-----------------------------------------------------------------

And this bit: 

Jan,... Here is another section, you conveniently cut out. Maybe you
should explain why you cut out this section? Was it embarrassing to you?
I mean, your statement is sort of correct,... but it shows a basic
misunderstanding of filesystems.

> > With compression there is a pretty high probability that one corrupted
> > byte or disk block will result in loss of a considerably larger amount
> > of data. 
> 
> Bad blocks are NOT dealt with by the filesystem,... so your comment is
> irrelevant, or just plain wrong.
> 
> If your filesystem is writing to bad blocks, then throw away your
> operating system.

-----------------------------------------------------------------

It is true that I considered your "most useful information," an
irrelevant section, which is why it was cut out (ignored).

I did not see my doing this, any worse than you doing it. I did not
realize that you were be so impolite.

As to your email being private, I had thought I had joined a mailing
list. I had not idea your email was meant to be private and just
considered it like all the others.

Now you mention it, I wondered why the email did not automatically list
the mailing lists, as recipients, and I had to add them. If I had
realized this I may have added the mailing lists as recipients, anyway.
It would be like me to do such. However, you should understand that I am
new to mailing lists.
 
> 
> > > Do you really have to repeat the results in every email you sent?
> > 
> > Damn, I did it again. WHY DO YOU CARE?
> 
> Because I saw them the first time around. And although the performance
> difference on those micro benchmarks seems quite impressive, I'm not
> convinced.

So, likewise, I saw your comments (you know the ones you miss so much)
the first time around, as I was not convinced of their worth.

The benchmarks measure certain data. Its fine you do not read into them,
stuff that isn't there, like reliability, for example. 

> > Look, its simple, I am (among other things) discussing these results, so
> > people need to see them.
> 
> However, you do not discuss, you just repeat, and repeat, and repeat.

I never said I wanted discussion, you just said I did.

You just repeat, and repeat, and repeat.

In reality, I quite appreciate reasonable discussion. But, I doubt I
will get much from you.

> But for what reason. Do you want an actual discussion, or do you hate
> the reiserfs developers so much that you want to antagonize any and all
> other Linux file systems developers?

Why do you think I hate reiserfs developers? That is an insane claim.
Why would I hate reiser3 developers?
Why would I hate reiser4 developers? 
Why would I even dislike them?

I think Hans Reiser is a genius. Is that what you mean by hate?

Answer this question. Why do YOU think I am antagonizing reiserfs
developers?

You must have a reason for stating what you have.

> > By the way, I have pulled the plug on my REISER4 system, a number of
> > times now, and it recovers without problem.
> 
> Very nice for you. I guess you have also not been hit by out of memory
> conditions or failing partial writes. So I guess we can ignore the patch
> that was just sent a day or two ago to the mailing list because you
> succesfully pulled the plug, a number of times at that.

Why are you attacking me with sarcasm, when I have just stated a simple
fact?

> > > > I have set up a Reiser4 partition with gzip compression, here is the
> > > > difference in disk usage of a typical Debian installation on two 10GB
> > > > partitions, one with Reiser3 and the other with Reiser4.
> > > > 
> > > > debian:/# df
> > > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > > /dev/sda3             10490104   6379164   4110940  61% /3
> > > > /dev/sda7              9967960   2632488   7335472  27% /7
> > > ...
> > > > Partition 3 is Reiser3 -- uses 6.4 GB.
> > > > Partition 7 is Reiser4 -- uses 2.6 GB.
> > > > 
> > > > So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
> > > > 6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
> > > > info).
> > > 
> > > Wow, consider me totally and completely, unimpressed.
> > > 
> 
> Here is the part of my email that you seemed to totally ignore,

Yes, I ignored it,.... is that a crime?

> You've just saved yourself $3.80, now go get yourself a latte.
> (see. http://tomayko.com/weblog/2006/09/11/that-dilbert-cartoon)
> 
> Seriously, disk storage is getting less expensive all the time, you can
> already buy a 250GB SATA drive for $70. Also, compression doesn't help
> once you store already compressed data such as jpeg images, mp3 files,
> or mpeg2/4/divx video files. Not only are the savings non-existant, but
> we still end up with the disadvantage that corruption propagates to
> multiple files because of the compression in the file system.

I believe that compression in REISER4 is just a wrapper for gzip or lzo.
So each file is compressed and your "corruption propagation" is just in
your imagination.

> And if it doesn't propagate across multiple files, the compression can't
> be all that good 

It can be as good as gzipping every file, ie, pretty damn good.

> since it can't benefit as much from similarity between
> files. So if that is the case and you really want to save diskspace you
> almost have to look at read-only compressed filesystems such as cramfs,
> squashfs, zisofs, cloop and various other variants in combination with
> a unionfs overlay to get read/write functionality.
> 
> But in the end everything is a tradeoff. You can save diskspace, but
> increase the cost of corruption. 

You deliberately ignored the fact that bad blocks are NOT dealt with by
the filesystem,... but by the operating system. Like I said: If your
filesystem is writing to bad blocks, then throw away your operating
system.

> Use a better compression algorithm, but
> that uses more CPU or which is visible in performance of the
> application. 

Not necessarily, just look at the bonnie++ tests. The tests show that
using compression (with todays hardware) leads to read speeds that are
some FOUR times the physical disk read rate,... 

but you ignore this as you have some strange anti-REISER religion.

Think about it,... read speeds that are some FOUR times the physical
disk read rate,... impossible without the use of compression (or
something similar).

> This can be offset by caching more, and being lazier with
> writebacks, but that hurts on-disk consistency, creates more memory
> pressure (more swapout/paging) and generally slows down other
> applications that aren't actually accessing the disk. Having a fast
> multi-core cpu and lots of memory helps a lot, but at some point what
> tradeoff did we just make, we saved a couple of dollars not having to
> buy a larger disk, but we spend considerably more on the more
> expensive cpu and memory.
> 

Wow, consider me STILL totally impressed by your AMAZING BIAS.
Would you like to tell me why you are SO BIASED against REISER4?

John.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Choose from over 50 domains or use your own

Re: Reiser4. BEST FILESYSTEM EVER.

[Willy Tarreau]

On Fri, Apr 06, 2007 at 10:58:45PM -0700, johnrobertbanks@fastmail.fm wrote:
> You know,... you cut out this bit:
> 
> -----------------------------------------------------------------
> 
> > The following benchmarks are from
> > 
> > http://linuxhelp.150m.com/resources/fs-benchmarks.htm or,
> > http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

...

Hey John, please change your disk, it's scratched and you're repeating
yourself again and again. At first I thought "Oh cool, some good news
about reiser4", now when I see "reiserfs" in a thread, I think "oh no,
not this boring guy who escaped from the asylum again !". I hope this
thread will be cut shortly so that you stop doing bad publicity to
reiserfs and its developers, because when a product is indicated as
good by stupid people, it's really doing harm.

Also, about this part :
[Jan]
> > But in the end everything is a tradeoff. You can save diskspace, but
> > increase the cost of corruption. 

I don't 100% agree with Jan, because for some usages (temporary space),
light compression can increase speed. For instance, when processing logs,
I get better speed by compressing intermediate files with LZO on the fly.

[John]
> You deliberately ignored the fact that bad blocks are NOT dealt with by
> the filesystem,... but by the operating system. Like I said: If your
> filesystem is writing to bad blocks, then throw away your operating
> system.

But what you write here is complete crap. The filesystem relies on a
linear block device. The operating system is responsible for doing
read retries or reporting errors on bad blocks, but the FS and only
the FS can decide how not to use some known defective areas, for
instance not putting any metadata on them nor any useful data.

Now if you want to stop writing stupid things again and again, take
your bag, don't miss the bus to school, and listen to the teachers
instead of playing games on your calculator.

Willy
PS: non need to reply either, I'll kill this thread and your address here.

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Hi Willy,...

> With decent CPU, you can reach higher read/write data rates than what a
> single off-the-shelf disk can achieve. For this reason, I think that
> reiser4 would be worth trying for this particular usage.

Glad to see you are willing to give Reiser4 a go.

Good man.

----------------------------------------------------------------------


On Sat, 7 Apr 2007 09:15:35 +0200, "Willy Tarreau" <w@1wt.eu> said:
> On Fri, Apr 06, 2007 at 10:58:45PM -0700, johnrobertbanks@fastmail.fm
> wrote:
> > You know,... you cut out this bit:
> > 
> > -----------------------------------------------------------------
> > 
> > > The following benchmarks are from
> > > 
> > > http://linuxhelp.150m.com/resources/fs-benchmarks.htm or,
> > > http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> ...
> 
> Hey John, please change your disk, it's scratched and you're repeating
> yourself again and again. At first I thought "Oh cool, some good news
> about reiser4", now when I see "reiserfs" in a thread, I think "oh no,
> not this boring guy who escaped from the asylum again !". I hope this
> thread will be cut shortly so that you stop doing bad publicity to
> reiserfs and its developers, because when a product is indicated as
> good by stupid people, it's really doing harm.
> 
> Also, about this part :
> [Jan]
> > > But in the end everything is a tradeoff. You can save diskspace, but
> > > increase the cost of corruption. 
> 
> I don't 100% agree with Jan, because for some usages (temporary space),
> light compression can increase speed. For instance, when processing logs,
> I get better speed by compressing intermediate files with LZO on the fly.
> 
> [John]
> > You deliberately ignored the fact that bad blocks are NOT dealt with by
> > the filesystem,... but by the operating system. Like I said: If your
> > filesystem is writing to bad blocks, then throw away your operating
> > system.
> 
> But what you write here is complete crap. The filesystem relies on a
> linear block device. The operating system is responsible for doing
> read retries or reporting errors on bad blocks, but the FS and only
> the FS can decide how not to use some known defective areas, for
> instance not putting any metadata on them nor any useful data.
> 
> Now if you want to stop writing stupid things again and again, take
> your bag, don't miss the bus to school, and listen to the teachers
> instead of playing games on your calculator.
> 
> Willy
> PS: non need to reply either, I'll kill this thread and your address
> here.
> 
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - One of many happy users:
  http://www.fastmail.fm/docs/quotes.html

Re: Reiser4. BEST FILESYSTEM EVER.

[Krzysztof Halasa]

johnrobertbanks@fastmail.fm writes:

> Why do you think I hate reiserfs developers? That is an insane claim.
> Why would I hate reiser3 developers?
> Why would I hate reiser4 developers? 
> Why would I even dislike them?
>
> I think Hans Reiser is a genius. Is that what you mean by hate?

I think they could hire a person with a bit better marketing skills,
though. People on a technical mailing list don't buy things just
because something on TV told them they have to.

> Answer this question. Why do YOU think I am antagonizing reiserfs
> developers?

That might be just a side effect.

> Think about it,... read speeds that are some FOUR times the physical
> disk read rate,... impossible without the use of compression (or
> something similar).

It's really impossible with compression only unless you're writing
only zeros or stuff alike. I don't know what bonnie uses for testing
but real life data doesn't compress 4 times. Two times, sometimes,
but then it will be typically slower than disk access (I mean read,
as write will be much slower).

You can get faster I/O (both linear speed and access times) using
multiple disks (mirrors etc). Perhaps some ZFS ideas would do us
some good?

Gzip - 3 files (zeros only, raw DV data from video camera, x86_64
kernel rpm file), 10 MB of data (10*1024*1024), done on tmpfs so no
real disk speed factor. The CPU is AMD64 with 1 MB cache per core,
2600 MHz clock (clock scaling disabled). That's my typical usage
pattern (well, not counting these zeros).

$ l -Ggh zeros dv bin
-rw-r--r-- 1 10M Apr  7 15:30 bin
-rw-r--r-- 1 10M Apr  7 15:31 dv
-rw-r--r-- 1 10M Apr  7 15:31 zeros

$ for f in zeros dv bin; do time gzip $f; done
real    0m0.112s
real    0m0.686s
real    0m0.559s

Dealing with pure zeros gzip can get almost 90 MB/s compressing, but
with DV and rpm it only does 14.5 and almost 18 MB/s respectively...

$ l -Ggh zeros.gz dv.gz bin.gz
-rw-r--r-- 1  10K Apr  7 15:31 zeros.gz
-rw-r--r-- 1 9.1M Apr  7 15:31 dv.gz
-rw-r--r-- 1 9.3M Apr  7 15:30 bin.gz

... and though the numbers may still sound impressive, space savings
are less than 10%.

$ for f in zeros dv bin; do time gunzip $f.gz; done
real    0m0.067s
real    0m0.131s
real    0m0.120s

Decompression gives 150 MB/s for zeros and ~ 80 MB/s for DV and rpm.

$ for f in zeros dv bin; do time gzip -1 $f; done
real    0m0.079s
real    0m0.572s
real    0m0.530s

Supposed to be "fastest gzip". 126 MB/s for zeros but still less than
19 MB/s for DV and rpm.

$ l -Ggh zeros.gz dv.gz bin.gz
-rw-r--r-- 1  45K Apr  7 15:31 zeros.gz
-rw-r--r-- 1 9.2M Apr  7 15:31 dv.gz
-rw-r--r-- 1 9.3M Apr  7 15:30 bin.gz

$ for f in zeros dv bin; do time gunzip $f.gz; done
real    0m0.044s
real    0m0.135s
real    0m0.120s

It seems gzip can decompress zeros with 227 MB/s rate.
I assume the "4x read speed" claim comes from something like this.

$ /sbin/hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:  210 MB in  3.02 seconds =  69.59 MB/sec

$ echo "69.59*4" | bc
278.36

Seems you'd need a faster algorithm, faster machine or slower disk
- slower than this cheap SATA with disabled NCQ (NV SATA) at least:

$ cat /sys/block/sda/device/model
Maxtor 6V250F0

Please note that aplication-level compression usually gives way
better results - the application knows much more.
-- 
Krzysztof Halasa

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Krzysztof -- Aren't you missing the point? Twice the speed would be
great,... even a 50% increase,... even a 0% increase.

I checked what bonnie++ actually writes to its test files, for you. It
is about 98-99% zeros.

Still, the results record sequential reads, of 232,729 K/sec, nearly
four times the physical disk read rate, 63,160 K/sec, of the hard drive.

The sequential writes are about three times the physical disk write
rate.

Even if the speed increase was zero, the more efficient use of disk
space means that Reiser4 is worth investigating.

People use RAID arrays to achieve speed increases. 

The people who developed RAID clearly thought that increases in speed
were worth investigating.

> 
> > Why do you think I hate reiserfs developers? That is an insane claim.
> > Why would I hate reiser3 developers?
> > Why would I hate reiser4 developers? 
> > Why would I even dislike them?
> >
> > I think Hans Reiser is a genius. Is that what you mean by hate?
> 
> I think they could hire a person with a bit better marketing skills,
> though. People on a technical mailing list don't buy things just
> because something on TV told them they have to.

I don't work for Reiser if that is what you are suggesting.

And people buy all sorts of lies because someone on TV told them it was
true.

Did you believe Iraq had WMD (weapons of mass destruction) because a
bunch of American liars told you this on TV? Millions of Americans did.

> > Answer this question. Why do YOU think I am antagonizing reiserfs
> > developers?
> 
> That might be just a side effect.
> 
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Send your email first class

Re: Reiser4. BEST FILESYSTEM EVER.

[Pekka Enberg]

On 4/7/07, johnrobertbanks@fastmail.fm <johnrobertbanks@fastmail.fm> wrote:
> I checked what bonnie++ actually writes to its test files, for you. It
> is about 98-99% zeros.
>
> Still, the results record sequential reads, of 232,729 K/sec, nearly
> four times the physical disk read rate, 63,160 K/sec, of the hard drive.

Excellent! You've established the undeniable hard cold fact that
reiser4 beats the crap out of all other filesystems, when the files
are 98-99% filled with zeros. You've proven your point, so can we stop
this thread now?

Re: Reiser4. BEST FILESYSTEM EVER.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

On Sat, 07 Apr 2007 16:11:46 +0200, Krzysztof Halasa said:

> > Think about it,... read speeds that are some FOUR times the physical
> > disk read rate,... impossible without the use of compression (or
> > something similar).
> 
> It's really impossible with compression only unless you're writing
> only zeros or stuff alike. I don't know what bonnie uses for testing
> but real life data doesn't compress 4 times. Two times, sometimes,

All depends on your data.  From a recent "compress the old logs" job on
our syslog server:

/logs/lennier.cc.vt.edu/2007/03/maillog-2007-0308:       85.4% -- replaced with /logs/lennier.cc.vt.edu/2007/03/maillog-2007-0308.gz

And it wasn't a tiny file either - it's a busy mailserver, the logs run to
several hundred megabytes a day.  Syslogs *often* compress 90% or more,
meaning a 10X compression.

> but then it will be typically slower than disk access (I mean read,
> as write will be much slower).

Actually, as far back as 1998 or so, I was able to document 20% *speedups*
on an AIX system that supported compressed file systems - and that was from
when a 133mz PowerPC 604e was a *fast* machine.   Since then, CPUs have gotten
faster at a faster rate than disks have, even increasing the speedup.

The basic theory is that unless you're sitting close to 100%CPU, it is *faster*
to burn some CPU to compress/decompress a 4K chunk of data down to 2K, and then
move 2K to the disk drive, than it is to move 4K.  It's particularly noticable
for larger files - if you can apply the compression to  remove the need to move
2M of data faster than you can move 2M of data, you win.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Reiser4. BEST FILESYSTEM EVER.

[Adrian Bunk]

On Sat, Apr 07, 2007 at 01:10:31PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Sat, 07 Apr 2007 16:11:46 +0200, Krzysztof Halasa said:
> 
> > > Think about it,... read speeds that are some FOUR times the physical
> > > disk read rate,... impossible without the use of compression (or
> > > something similar).
> > 
> > It's really impossible with compression only unless you're writing
> > only zeros or stuff alike. I don't know what bonnie uses for testing
> > but real life data doesn't compress 4 times. Two times, sometimes,
> 
> All depends on your data.  From a recent "compress the old logs" job on
> our syslog server:
> 
> /logs/lennier.cc.vt.edu/2007/03/maillog-2007-0308:       85.4% -- replaced with /logs/lennier.cc.vt.edu/2007/03/maillog-2007-0308.gz
> 
> And it wasn't a tiny file either - it's a busy mailserver, the logs run to
> several hundred megabytes a day.  Syslogs *often* compress 90% or more,
> meaning a 10X compression.
> 
> > but then it will be typically slower than disk access (I mean read,
> > as write will be much slower).
> 
> Actually, as far back as 1998 or so, I was able to document 20% *speedups*
> on an AIX system that supported compressed file systems - and that was from
> when a 133mz PowerPC 604e was a *fast* machine.   Since then, CPUs have gotten
> faster at a faster rate than disks have, even increasing the speedup.
> 
> The basic theory is that unless you're sitting close to 100%CPU, it is *faster*
> to burn some CPU to compress/decompress a 4K chunk of data down to 2K, and then
> move 2K to the disk drive, than it is to move 4K.  It's particularly noticable
> for larger files - if you can apply the compression to  remove the need to move
> 2M of data faster than you can move 2M of data, you win.

Counterpoints:
- not only CPUs have became faster, RAM has become faster, too
  a kernel tree after an allyesconfig build is at about 1 GB which is 
  less than half the size of RAM in my desktop computer
  if all disk accesses are asynchronous write accesses without any 
  pressure of being done quickly, compression can't improve performance
- today, much of the bigger data is already compressed data like mp3s
  or movies
- for cases like logfiles or databases, application specific compression
  should give best results

There might be special cases where compressed filesystems make sense, 
but my impression is that filesystem compresssion is not important and 
suited for current average systems.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Reiser4. BEST FILESYSTEM EVER.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

On Sat, 07 Apr 2007 16:11:46 +0200, Krzysztof Halasa said:
>
> Gzip - 3 files (zeros only, raw DV data from video camera, x86_64
> kernel rpm file), 10 MB of data (10*1024*1024),
> $ l -Ggh zeros dv bin
> -rw-r--r-- 1 10M Apr  7 15:30 bin
> -rw-r--r-- 1 10M Apr  7 15:31 dv
> -rw-r--r-- 1 10M Apr  7 15:31 zeros

> $ l -Ggh zeros.gz dv.gz bin.gz
> -rw-r--r-- 1  10K Apr  7 15:31 zeros.gz
> -rw-r--r-- 1 9.1M Apr  7 15:31 dv.gz
> -rw-r--r-- 1 9.3M Apr  7 15:30 bin.gz
> 
> ... and though the numbers may still sound impressive, space savings
> are less than 10%.

I am quite sure that the kernel RPM file is *already* compressed, at least
somewhat.  Otherwise, it's hard to explain this:

-rw-r--r--    1 529      263     17835757   Apr  5 00:19   kernel-2.6.20-1.3045.fc7.x86_64.rpm

% du -s /lib/modules/2.6.20-1.3038.fc7/
76436   /lib/modules/2.6.20-1.3038.fc7/

and it can't all be slack space at ends of files:

% find /lib/modules/2.6.20-1.3038.fc7/ -type f | wc -l
1482

Even on a 4K filesystem, the *max* wasted slack would be about 4M.

And what do you know - if you tar.gz that /lib/modules:

% tar czf /tmp/kern.tar.gz /lib/modules/2.6.20-1.3038.fc7/
tar: Removing leading `/' from member names
% ls -l /tmp/kern.tar.gz 
-rw-r--r-- 1 valdis valdis 15506359 2007-04-07 13:19 /tmp/kern.tar.gz

The *compressed* tar is about 15M (remember the .rpm contained a 2M vmlinuz
as well - that;s compressed too).  So we're right up to the 17M of the .rpm,
which indicates that the RPM is compressed at a factor close to tar.gz.

I'd not be surprised to find out that your digital-video also contains
at least some light compression - if it's mpeg or similar, that's already
had some *heavy* compression done to it....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Reiser4. BEST FILESYSTEM EVER.

[Krzysztof Halasa]

Valdis.Kletnieks@vt.edu writes:

> I am quite sure that the kernel RPM file is *already* compressed, at least
> somewhat.

Sure - that's the point - it's better to have the tool compress
data when it makes sense.

OTOH I think Reiser4 fs is not about transparent compression, it's
rather about the plugins etc. There are other filesystems with
transparent compression, that's nothing new.
-- 
Krzysztof Halasa

Re: Reiser4. BEST FILESYSTEM EVER.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

On Fri, 06 Apr 2007 19:47:36 PDT, johnrobertbanks@fastmail.fm said:
> On Fri, 6 Apr 2007 11:21:19 -0400, "Jan Harkes" <jaharkes@cs.cmu.edu>

> > With compression there is a pretty high probability that one corrupted
> > byte or disk block will result in loss of a considerably larger amount
> > of data. 
> 
> Bad blocks are NOT dealt with by the filesystem,... so your comment is
> irrelevant, or just plain wrong.
> 
> If your filesystem is writing to bad blocks, then throw away your
> operating system.

You know... occasionally, blocks go bad *after* you write to them.  If
you have an uncompressed filesystem, it's often possible to recover most
of the file , and just have a few 512-byte blocks of zeros, simply by
doing something like 'dd if=bad.file of=bad.file bs=512 conv=noerror'
or careful applications of 'skip=N'.  If it's compressed, you usually
can't recover the rest of a compression group if a previous block is lost.

(And for those who talk about backups - yes, taking backups is good.
However, it's the rare laptop or desktop machine that can afford the
luxury of RAID disks, and backups usually happen once a night, if that
often.  This means that if you've been working hard on something important
all day, and the disk blows chunks at 4:30PM, you *will* be suddenly very
concerned over exactly how much you can recover off the failing drive....

And yes, I'd *love* to have all my users connected to nice SAN systems that do
snapshotting and remote replication to DR sites and all that - but have you
ever *priced* a petabyte of SAN storage, the NAS gateways to serve it to users,
and upgrading several tens of thousands of network ports to Gig-E? Hint -
US$1M would get us through a pilot, and probably $5M and up to *start*
deployment. Anybody wanna buy us an EMC DMX-3? :)

http://www.emc.com/products/systems/symmetrix/DMX_series/DMX3.jsp


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Reiser4. BEST FILESYSTEM EVER.

[Lennart Sorensen]

On Thu, Apr 05, 2007 at 09:32:11PM -0700, johnrobertbanks@fastmail.fm wrote:
> Don't you agree, that "If they are accurate,.... THEN they are obviously
> very relevant."

Nope, if they are accurate and they have something to do with your
particular usage and applications, then they are relevant.  But it
requires both to make them relevant.  Although it may be possible for a
benchmark to be relevant even if not particularly accurate.

> I have set up a Reiser4 partition with gzip compression, here is the
> difference in disk usage of a typical Debian installation on two 10GB
> partitions, one with Reiser3 and the other with Reiser4.
> 
> debian:/# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda3             10490104   6379164   4110940  61% /3
> /dev/sda7              9967960   2632488   7335472  27% /7
> 
> Partitions 3 and 7 have exactly the same data on them (the typical
> Debian install).
> 
> The partitions are exactly the same size (although df records different
> sizes).
> 
> Partition 3 is Reiser3 -- uses 6.4 GB.
> Partition 7 is Reiser4 -- uses 2.6 GB.
> 
> So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
> 6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
> info).
> 
> Don't you think this result is significant in itself?

Only if you think disk space is so valuable that trading cpu time to
compress and decompress the data is a good trade off.  It is not one I
would want to make.  So you saved 3GB, what is that?  About $1 worth?
maybe $2 if you have raid.  How much extra time and cpu will it take to
access the data that way?  How much extra electricity will the cpu use?
What is your time worth?  There are so many variables.  Do you _trust_
reiserfs4 to not loose your data any more or less than some other
filesystem?

--
Len Sorensen

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Lennart. Tell me again that these results from 

http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

are not of interest to you. I still don't understand why you have your
head in the sand.

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.


Column one measures the time taken to complete the bonnie++ benchmarking
test (run with the parameters bonnie++ -n128:128k:0)

Column two, Disk Usage: measures the amount of disk used to store 655MB
of raw data (which was 3 different copies of the Linux kernel sources).

OR LOOK AT THE FULL RESULTS:

.-------------------------------------------------.
|File         |Disk |Copy |Copy |Tar  |Unzip| Del |
|System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
|Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
.-------------------------------------------------.
|REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
|REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
|REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
|REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
|NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
|NTFS         | 779 | 781 | 173 |   X |   X |   X |
|REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
|XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
|JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
|EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
|EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
|EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
|EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
|FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
.-------------------------------------------------.


Each test was preformed 5 times and the average value recorded.
Disk Usage: The amount of disk used to store the data (which was 3
different copies of the Linux kernel sources).
The raw data (without filesystem meta-data, block alignment wastage,
etc) was 655MB.
Copy 655MB (1): Copy the data over a partition boundary.
Copy 655MB (2): Copy the data within a partition.
Tar Gzip 655MB: Tar and Gzip the data.
Unzip UnTar 655MB: UnGzip and UnTar the data.
Del 2.5 Gig: Delete everything just written (about 2.5 Gig).

To get a feel for the performance increases that can be achieved by
using compression, we look at the total time (in seconds) to run the
test:

bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)

.-------------------.
| FILESYSTEM | TIME |
.-------------------.
|REISER4 lzo |  1938|
|REISER4 gzip|  2295|
|REISER4     |  3462|
|EXT4        |  4408|
|EXT2        |  4092|
|JFS         |  4225|
|EXT3        |  4421|
|XFS         |  4625|
|REISER3     |  6178|
|FAT32       | 12342|
|NTFS-3g     |>10414|
.-------------------.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Access all of your messages and folders
                          wherever you are

Re: Reiser4. BEST FILESYSTEM EVER.

[Lennart Sorensen]

On Sat, Apr 07, 2007 at 05:44:57PM -0700, johnrobertbanks@fastmail.fm wrote:
> Lennart. Tell me again that these results from 
> 
> http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

Hmm, copying kernel sources around.  Not that interesting.  How does it
handle mpeg2 files (the majority of my personal data files is on a
mythtv system).  So a few large files, with mostly linear access, and
the occational file deletion.  Compression would gain nothing.

> are not of interest to you. I still don't understand why you have your
> head in the sand.

Well I find it hard to get excited about new filesystems.  I had
sufficiently nasty data loses due to reiserfs 3 back in the early 2.4
kernel days, that I no longer get excited about new filesystems.  now I
want something I trust that hasn't destroyed any of my data.  I tried
XFS for a while, ut the early 2.6 kernels had some nasty bugs in XFS too
that made that pretty much unusable.  Now I just stick with ext3.  Screw
performance, give me something that works all the time.

> .-------------------------.
> | FILESYSTEM | TIME |DISK |
> | TYPE       |(secs)|USAGE|
> .-------------------------.
> |REISER4 lzo | 1938 | 278 |
> |REISER4 gzip| 2295 | 213 |
> |REISER4     | 3462 | 692 |
> |EXT2        | 4092 | 816 |
> |JFS         | 4225 | 806 |
> |EXT4        | 4408 | 816 |
> |EXT3        | 4421 | 816 |
> |XFS         | 4625 | 779 |
> |REISER3     | 6178 | 793 |
> |FAT32       |12342 | 988 |
> |NTFS-3g     |10414 | 772 |
> .-------------------------.
> 
> 
> Column one measures the time taken to complete the bonnie++ benchmarking
> test (run with the parameters bonnie++ -n128:128k:0)

Time without cpu usage is not interesting.  If you can increase
filesystem speed by 10% by doubling cpu load, then I don't want the
increase.  It is all relative.  Wall clock time by itself just doesn't
contain enough data to be useful.

> Column two, Disk Usage: measures the amount of disk used to store 655MB
> of raw data (which was 3 different copies of the Linux kernel sources).

I remember disk compression from the DOS days.  Disk space is too cheap
to bother with that crap anymore.  I don't care if it can theoretically
turn idle cpu cycles into improved disk speed.  Sometimes I don't have
idle cpu cycles to waste on that.

> OR LOOK AT THE FULL RESULTS:
> 
> .-------------------------------------------------.
> |File         |Disk |Copy |Copy |Tar  |Unzip| Del |
> |System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
> |Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
> .-------------------------------------------------.
> |REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
> |REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
> |REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
> |REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
> |NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
> |NTFS         | 779 | 781 | 173 |   X |   X |   X |
> |REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
> |XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
> |JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
> |EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
> |EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
> |EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
> |EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
> |FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
> .-------------------------------------------------.
> 
> 
> Each test was preformed 5 times and the average value recorded.
> Disk Usage: The amount of disk used to store the data (which was 3
> different copies of the Linux kernel sources).
> The raw data (without filesystem meta-data, block alignment wastage,
> etc) was 655MB.
> Copy 655MB (1): Copy the data over a partition boundary.
> Copy 655MB (2): Copy the data within a partition.
> Tar Gzip 655MB: Tar and Gzip the data.
> Unzip UnTar 655MB: UnGzip and UnTar the data.
> Del 2.5 Gig: Delete everything just written (about 2.5 Gig).
> 
> To get a feel for the performance increases that can be achieved by
> using compression, we look at the total time (in seconds) to run the
> test:

kernel sources are some of the most compressable data files around.  Try
with some interesting data instead, like something with larger files,
mostly binary, which isn't likely to compess very much.

> bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)
> 
> .-------------------.
> | FILESYSTEM | TIME |
> .-------------------.
> |REISER4 lzo |  1938|
> |REISER4 gzip|  2295|
> |REISER4     |  3462|
> |EXT4        |  4408|
> |EXT2        |  4092|
> |JFS         |  4225|
> |EXT3        |  4421|
> |XFS         |  4625|
> |REISER3     |  6178|
> |FAT32       | 12342|
> |NTFS-3g     |>10414|
> .-------------------.
> -- 

Well Reiser4 certainly looks impresive, but I still want to know what the
cpu load is like, what the repair tools are like, how well it handles
power failures in the middle of a write (I didn't like the way reiser3
would claim to have the filesystem back to a sane state, but the file
it had been in the middle of writing now contained garbage in many parts
of it, with no warning what so ever that this had taken place).

Performance looks impresive, but based on the history of reiser3 there
is a lot of historical damage to undo before I will even consider
testing it out.

--
Len Sorensen

Re: Reiser4. BEST FILESYSTEM EVER.

[Theodore Tso]

On Sat, Apr 07, 2007 at 05:44:57PM -0700, johnrobertbanks@fastmail.fm wrote:
> To get a feel for the performance increases that can be achieved by
> using compression, we look at the total time (in seconds) to run the
> test:

You mean the performance increases of writing a file which is mostly
all zero's?  Yawn.....

					- Ted

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Teddy,

It is a pity you don't address the full set of results, when you make
your snide comments.

Now since you have them,... why don't you make reasoned comment about
them.

You can read more here:

http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm

.-------------------------.
| FILESYSTEM | TIME |DISK |
| TYPE       |(secs)|USAGE|
.-------------------------.
|REISER4 lzo | 1938 | 278 |
|REISER4 gzip| 2295 | 213 |
|REISER4     | 3462 | 692 |
|EXT2        | 4092 | 816 |
|JFS         | 4225 | 806 |
|EXT4        | 4408 | 816 |
|EXT3        | 4421 | 816 |
|XFS         | 4625 | 779 |
|REISER3     | 6178 | 793 |
|FAT32       |12342 | 988 |
|NTFS-3g     |10414 | 772 |
.-------------------------.


Column one measures the time taken to complete the bonnie++ benchmarking
test (run with the parameters bonnie++ -n128:128k:0)

Column two, Disk Usage: measures the amount of disk used to store 655MB
of raw data (which was 3 different copies of the Linux kernel sources).

OR LOOK AT THE FULL RESULTS:

.-------------------------------------------------.
|File         |Disk |Copy |Copy |Tar  |Unzip| Del |
|System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
|Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
.-------------------------------------------------.
|REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
|REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
|REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
|REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
|NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
|NTFS         | 779 | 781 | 173 |   X |   X |   X |
|REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
|XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
|JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
|EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
|EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
|EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
|EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
|FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
.-------------------------------------------------.


Each test was preformed 5 times and the average value recorded.
Disk Usage: The amount of disk used to store the data (which was 3
different copies of the Linux kernel sources).
The raw data (without filesystem meta-data, block alignment wastage,
etc) was 655MB.
Copy 655MB (1): Copy the data over a partition boundary.
Copy 655MB (2): Copy the data within a partition.
Tar Gzip 655MB: Tar and Gzip the data.
Unzip UnTar 655MB: UnGzip and UnTar the data.
Del 2.5 Gig: Delete everything just written (about 2.5 Gig).


To get a feel for the performance increases that can be achieved by
using compression, we look at the total time (in seconds) to run the
test:

bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)

.-------------------.
| FILESYSTEM | TIME |
.-------------------.
|REISER4 lzo |  1938|
|REISER4 gzip|  2295|
|REISER4     |  3462|
|EXT4        |  4408|
|EXT2        |  4092|
|JFS         |  4225|
|EXT3        |  4421|
|XFS         |  4625|
|REISER3     |  6178|
|FAT32       | 12342|
|NTFS-3g     |>10414|
.-------------------.



On Sat, 7 Apr 2007 22:56:32 -0400, "Theodore Tso" <tytso@mit.edu> said:
> On Sat, Apr 07, 2007 at 05:44:57PM -0700, johnrobertbanks@fastmail.fm
> wrote:
> > To get a feel for the performance increases that can be achieved by
> > using compression, we look at the total time (in seconds) to run the
> > test:
> 
> You mean the performance increases of writing a file which is mostly
> all zero's?  Yawn.....
> 
> 					- Ted
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - IMAP accessible web-mail

Re: Reiser4. BEST FILESYSTEM EVER.

[Jose Celestino]

Words by johnrobertbanks@fastmail.fm [Sat, Apr 07, 2007 at 09:13:32PM -0700]:
> Teddy,
> 
> It is a pity you don't address the full set of results, when you make
> your snide comments.
> 
> Now since you have them,... why don't you make reasoned comment about
> them.
> 
> You can read more here:
> 

John,

it is not because you keep posting the same numbers over and over again
(or is this your new signature?) that they will be more substantiated
(hint: cpu usage).
Just more annoying each time.

I'll remember to use reiser4 for my
90-percent-zero-files-no-need-for-proven-robustness-and-plenty-of-cpu-to-spare
boxes. Thank you.

> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> .-------------------------.
> | FILESYSTEM | TIME |DISK |
> | TYPE       |(secs)|USAGE|
> .-------------------------.
> |REISER4 lzo | 1938 | 278 |
> |REISER4 gzip| 2295 | 213 |
> |REISER4     | 3462 | 692 |
> |EXT2        | 4092 | 816 |
> |JFS         | 4225 | 806 |
> |EXT4        | 4408 | 816 |
> |EXT3        | 4421 | 816 |
> |XFS         | 4625 | 779 |
> |REISER3     | 6178 | 793 |
> |FAT32       |12342 | 988 |
> |NTFS-3g     |10414 | 772 |
> .-------------------------.
> 
> 
> Column one measures the time taken to complete the bonnie++ benchmarking
> test (run with the parameters bonnie++ -n128:128k:0)
> 
> Column two, Disk Usage: measures the amount of disk used to store 655MB
> of raw data (which was 3 different copies of the Linux kernel sources).
> 
> OR LOOK AT THE FULL RESULTS:
> 
> .-------------------------------------------------.
> |File         |Disk |Copy |Copy |Tar  |Unzip| Del |
> |System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
> |Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
> .-------------------------------------------------.
> |REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
> |REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
> |REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
> |REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
> |NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
> |NTFS         | 779 | 781 | 173 |   X |   X |   X |
> |REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
> |XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
> |JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
> |EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
> |EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
> |EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
> |EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
> |FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
> .-------------------------------------------------.
> 
> 
> Each test was preformed 5 times and the average value recorded.
> Disk Usage: The amount of disk used to store the data (which was 3
> different copies of the Linux kernel sources).
> The raw data (without filesystem meta-data, block alignment wastage,
> etc) was 655MB.
> Copy 655MB (1): Copy the data over a partition boundary.
> Copy 655MB (2): Copy the data within a partition.
> Tar Gzip 655MB: Tar and Gzip the data.
> Unzip UnTar 655MB: UnGzip and UnTar the data.
> Del 2.5 Gig: Delete everything just written (about 2.5 Gig).
> 
> 
> To get a feel for the performance increases that can be achieved by
> using compression, we look at the total time (in seconds) to run the
> test:
> 
> bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)
> 
> .-------------------.
> | FILESYSTEM | TIME |
> .-------------------.
> |REISER4 lzo |  1938|
> |REISER4 gzip|  2295|
> |REISER4     |  3462|
> |EXT4        |  4408|
> |EXT2        |  4092|
> |JFS         |  4225|
> |EXT3        |  4421|
> |XFS         |  4625|
> |REISER3     |  6178|
> |FAT32       | 12342|
> |NTFS-3g     |>10414|
> .-------------------.
> 
> 
> 
> On Sat, 7 Apr 2007 22:56:32 -0400, "Theodore Tso" <tytso@mit.edu> said:
> > On Sat, Apr 07, 2007 at 05:44:57PM -0700, johnrobertbanks@fastmail.fm
> > wrote:
> > > To get a feel for the performance increases that can be achieved by
> > > using compression, we look at the total time (in seconds) to run the
> > > test:
> > 
> > You mean the performance increases of writing a file which is mostly
> > all zero's?  Yawn.....
> > 
> > 					- Ted
> -- 
>   
>   johnrobertbanks@fastmail.fm
> 
> -- 
> http://www.fastmail.fm - IMAP accessible web-mail
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Jose Celestino
----------------------------------------------------------------
http://www.msversus.org/     ; http://techp.org/petition/show/1
http://www.vinc17.org/noswpat.en.html
----------------------------------------------------------------
"And on the trillionth day, Man created Gods." -- Thomas D. Pate

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

Jose,
  since you clearly have nothing useful to say. Why don't you let Teddy
  talk for himself.


On Sun, 8 Apr 2007 13:48:11 +0100, "Jose Celestino" <japc@co.sapo.pt>
said:
> Words by johnrobertbanks@fastmail.fm [Sat, Apr 07, 2007 at 09:13:32PM
> -0700]:
> > Teddy,
> > 
> > It is a pity you don't address the full set of results, when you make
> > your snide comments.
> > 
> > Now since you have them,... why don't you make reasoned comment about
> > them.
> > 
> > You can read more here:
> > 
> 
> John,
> 
> it is not because you keep posting the same numbers over and over again
> (or is this your new signature?) that they will be more substantiated
> (hint: cpu usage).
> Just more annoying each time.
> 
> I'll remember to use reiser4 for my
> 90-percent-zero-files-no-need-for-proven-robustness-and-plenty-of-cpu-to-spare
> boxes. Thank you.
> 
> > http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> > 
> > .-------------------------.
> > | FILESYSTEM | TIME |DISK |
> > | TYPE       |(secs)|USAGE|
> > .-------------------------.
> > |REISER4 lzo | 1938 | 278 |
> > |REISER4 gzip| 2295 | 213 |
> > |REISER4     | 3462 | 692 |
> > |EXT2        | 4092 | 816 |
> > |JFS         | 4225 | 806 |
> > |EXT4        | 4408 | 816 |
> > |EXT3        | 4421 | 816 |
> > |XFS         | 4625 | 779 |
> > |REISER3     | 6178 | 793 |
> > |FAT32       |12342 | 988 |
> > |NTFS-3g     |10414 | 772 |
> > .-------------------------.
> > 
> > 
> > Column one measures the time taken to complete the bonnie++ benchmarking
> > test (run with the parameters bonnie++ -n128:128k:0)
> > 
> > Column two, Disk Usage: measures the amount of disk used to store 655MB
> > of raw data (which was 3 different copies of the Linux kernel sources).
> > 
> > OR LOOK AT THE FULL RESULTS:
> > 
> > .-------------------------------------------------.
> > |File         |Disk |Copy |Copy |Tar  |Unzip| Del |
> > |System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
> > |Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
> > .-------------------------------------------------.
> > |REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
> > |REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
> > |REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
> > |REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
> > |NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
> > |NTFS         | 779 | 781 | 173 |   X |   X |   X |
> > |REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
> > |XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
> > |JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
> > |EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
> > |EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
> > |EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
> > |EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
> > |FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
> > .-------------------------------------------------.
> > 
> > 
> > Each test was preformed 5 times and the average value recorded.
> > Disk Usage: The amount of disk used to store the data (which was 3
> > different copies of the Linux kernel sources).
> > The raw data (without filesystem meta-data, block alignment wastage,
> > etc) was 655MB.
> > Copy 655MB (1): Copy the data over a partition boundary.
> > Copy 655MB (2): Copy the data within a partition.
> > Tar Gzip 655MB: Tar and Gzip the data.
> > Unzip UnTar 655MB: UnGzip and UnTar the data.
> > Del 2.5 Gig: Delete everything just written (about 2.5 Gig).
> > 
> > 
> > To get a feel for the performance increases that can be achieved by
> > using compression, we look at the total time (in seconds) to run the
> > test:
> > 
> > bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)
> > 
> > .-------------------.
> > | FILESYSTEM | TIME |
> > .-------------------.
> > |REISER4 lzo |  1938|
> > |REISER4 gzip|  2295|
> > |REISER4     |  3462|
> > |EXT4        |  4408|
> > |EXT2        |  4092|
> > |JFS         |  4225|
> > |EXT3        |  4421|
> > |XFS         |  4625|
> > |REISER3     |  6178|
> > |FAT32       | 12342|
> > |NTFS-3g     |>10414|
> > .-------------------.
> > 
> > 
> > 
> > On Sat, 7 Apr 2007 22:56:32 -0400, "Theodore Tso" <tytso@mit.edu> said:
> > > On Sat, Apr 07, 2007 at 05:44:57PM -0700, johnrobertbanks@fastmail.fm
> > > wrote:
> > > > To get a feel for the performance increases that can be achieved by
> > > > using compression, we look at the total time (in seconds) to run the
> > > > test:
> > > 
> > > You mean the performance increases of writing a file which is mostly
> > > all zero's?  Yawn.....
> > > 
> > > 					- Ted
> > -- 
> >   
> >   johnrobertbanks@fastmail.fm
> > 
> > -- 
> > http://www.fastmail.fm - IMAP accessible web-mail
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -- 
> Jose Celestino
> ----------------------------------------------------------------
> http://www.msversus.org/     ; http://techp.org/petition/show/1
> http://www.vinc17.org/noswpat.en.html
> ----------------------------------------------------------------
> "And on the trillionth day, Man created Gods." -- Thomas D. Pate
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Does exactly what it says on the tin

Re: Reiser4. BEST FILESYSTEM EVER.

[Willy Tarreau]

On Sun, Apr 08, 2007 at 06:21:29AM -0700, johnrobertbanks@fastmail.fm wrote:
> Jose,
>   since you clearly have nothing useful to say. Why don't you let Teddy
>   talk for himself.

John,

You should first apply your own advice to yourself. Annoying everyone
with the exact same mail 10 times a day is really disserving to the
cause you pretend to defend. Please stop tainting reiser4's reputation,
because I suspect that it can do far more things than what you make it
look like. Its developers certainly need useful reports instead of a
mentally deficient's rant.

Now please call the nurse for your injection and go back to bed.

Thank you in advance,
Willy

Re: Reiser4. BEST FILESYSTEM EVER.

[Theodore Tso]

The reason why I ignore the tar+gzip tests is that in the past Hans
has rigged the test by using a tar ball which was generated by
unpacking a set of kernel sources on a reiser4 filesystem, and then
repacking them using tar+gzip.  The result was a tar file whose files
were optimally laid out so that reiser4 could insert them into the
filesystem b-tree without doing any extra work.

I can't say for sure whether or not this set of benchmarks has done
this (there's not enough information describing the benchmark setup),
but the sad fact of the matter is that people trying to pitch Reiser4
have generated for themselves a reputation for using rigged
benchmarks.  Hans's used of a carefully stacked and ordered tar file
(which is the same as stacking a deck of cards), and your repeated use
of the bonnee++ benchmarks despite being told that it is a meaningless
result given the fact that well, zero's compress very well and most
people are interested in storing a file of all zeros, has caused me to
look at any benchmarks cited by Reiser4 partisans with a very
jaundiced and skeptical eye.

Fortunately for you, it's not up to me whether or not Reiser4 makes it
into the kernel.  And if it works for you, hey, go wild.  You can
always patch it into your own kernel and encourage others to do the
same with respect to getting it tested and adopted.  My personal take
on it is that Reiser3, Reiser4 and JFS suffer the same problems, which
is to say they have a very small and limited development community,
and this was referenced in Novell's decision to drop Reiser3:

http://linux.wordpress.com/2006/09/27/suse-102-ditching-reiserfs-as-it-default-fs/

SuSE has deprecated Reiser3 *and* JFS, and I believe quite strongly it
is the failure of the organizations to attract a diverse development
community is ultimately what doomed them in the long term, both in
terms of support as the kernel migrated and new feature support.  It
is for that reason that Hans' personality traits that tend to drive
away those developers who would help them, beyond those that he hires,
is what has been so self-destructive to Reiser4.  Read the
announcement Jeff Mahoney from SUSE Labs again; he pointed out was
that reiser3 was getting dropped even though it performs better than
ext3 in some scenarios.  There are many other considerations, such as
a filesystem's robustness in case on-disk corruption, long term
maintenance as the kernel maintains, availability of developers to
provide bug fixes, how well the system performs on systems with
multiple cores/CPU's, etc.

						- Ted

Re: Reiser4. BEST FILESYSTEM EVER.

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theodore Tso wrote:
> The reason why I ignore the tar+gzip tests is that in the past Hans
> has rigged the test by using a tar ball which was generated by
> unpacking a set of kernel sources on a reiser4 filesystem, and then
> repacking them using tar+gzip.  The result was a tar file whose files
> were optimally laid out so that reiser4 could insert them into the
> filesystem b-tree without doing any extra work.
> 
> I can't say for sure whether or not this set of benchmarks has done
> this (there's not enough information describing the benchmark setup),
> but the sad fact of the matter is that people trying to pitch Reiser4
> have generated for themselves a reputation for using rigged
> benchmarks.  Hans's used of a carefully stacked and ordered tar file
> (which is the same as stacking a deck of cards), and your repeated use
> of the bonnee++ benchmarks despite being told that it is a meaningless
> result given the fact that well, zero's compress very well and most
> people are interested in storing a file of all zeros, has caused me to
> look at any benchmarks cited by Reiser4 partisans with a very
> jaundiced and skeptical eye.
> 
> Fortunately for you, it's not up to me whether or not Reiser4 makes it
> into the kernel.  And if it works for you, hey, go wild.  You can
> always patch it into your own kernel and encourage others to do the
> same with respect to getting it tested and adopted.  My personal take
> on it is that Reiser3, Reiser4 and JFS suffer the same problems, which
> is to say they have a very small and limited development community,
> and this was referenced in Novell's decision to drop Reiser3:
> 
> http://linux.wordpress.com/2006/09/27/suse-102-ditching-reiserfs-as-it-default-fs/
> 
> SuSE has deprecated Reiser3 *and* JFS, and I believe quite strongly it
> is the failure of the organizations to attract a diverse development
> community is ultimately what doomed them in the long term, both in
> terms of support as the kernel migrated and new feature support.  It
> is for that reason that Hans' personality traits that tend to drive
> away those developers who would help them, beyond those that he hires,
> is what has been so self-destructive to Reiser4.  Read the
> announcement Jeff Mahoney from SUSE Labs again; he pointed out was
> that reiser3 was getting dropped even though it performs better than
> ext3 in some scenarios.  There are many other considerations, such as
> a filesystem's robustness in case on-disk corruption, long term
> maintenance as the kernel maintains, availability of developers to
> provide bug fixes, how well the system performs on systems with
> multiple cores/CPU's, etc.

Those are all arguments I've made and still stand by, but I should
address one point that has been repeated fairly often. Novell _isn't_
dropping support for Reiser3 in any of our products. The change only
refers to the choice of a default file system. Most users don't care
about which file system they use, and those that do are still free to
choose reiser3 if they want it. We'll support it and I still have
patches under development to improve it.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFGGTHYLPWxlyuTD7IRAj0SAJ4txD5NoStOA4GFgkzcXDdE/Xf9ngCZATNL
QtyNTGbi6YFbNF71T5C9hTA=
=Emwr
-----END PGP SIGNATURE-----

Re: Reiser4. BEST FILESYSTEM EVER.

[Christer Weinigel]

johnrobertbanks@fastmail.fm writes:

> Lennart. Tell me again that these results from 
> 
> http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> are not of interest to you. I still don't understand why you have your
> head in the sand.

Oh, for fucks sake, stop sounding like a broken record.  You have
repeated the same totally meaningless statistics more times than I
care to count.  Please shut the fuck up.

As you discovered yourself (even though you seem to fail to understand
the significance of your discovery), bonnie writes files that consist
of mostly zeroes.  If your normal use cases consist of creating a
bunch of files containing zeroes, reiser4 with compression will do
great.  Just lovely.  Except that nobody sane would store a lot of
files containing zeroes, except an an excercize in mental
masturbation.  So the two bonnie benchmarks with lzo and gzip are
totally meaningless for any real life usages.

As for the amount of disk needed to store three kernel trees, the
figures you quote show that Reiser4 does tail combining where the tail
of multiple files are stored in one disk block.  A nice trick that
seems save you about 15% disk space compared to ext3.  Now you have to
realise what that means, it means that if the disk block containing
those tails (or any metadata pointing at that block) gets corrupted,
instead of just losing one disk block for one file, you will have lost
the tail for all the files sharing that disk block.  Depending on your
personal prioritites, saving 15% of the space may be worth the risk to
you, or maybe not.  Personally, for the only disk I'm short on space
on, I mostly store flac encoded images of my CD collection, and saving
2kByte out of every 300MByte disk simply doesn't make any difference,
and I much prefer a stable file system that I can trust not to lose my
data.  You might make different choices.

The same goes for just about every feature that you tout, it has its
advantages, and it has its disadvantages.  Doing compression on data
is great if the data you store is compressible, and sucks if it isn't.
Doing compression on each disk block and then packing multiple
compressed blocks into each physical disk block will probably save
some space if the data is compressible, but at the same time it means
that you will spend a lot of CPU (and cache footprint) compressing and
uncompressing that data.  On a single user system where the CPU is
mostly idle it might not make much of a difference, on a heavily
loaded multiuser system it might do.

Logs can be compressed quite well using a block based compression
scheme, but the logs can be compressed even better by doing
compression on the whole file with gzip.  So what's the best choice,
to do transparent compression on the fly giving ok compression or
teaching the userspace tools to do compression of old logs and get
really good compression?  Or maybe disk space really isn't that
important anyway and the best thing is to just leave the logs
uncompressed.

Another example: one of the things Reiser3 is supposed to be really
good for is storing an INN news spool, doing tail merging of lots of
individual files containing articles gives a great space saving, and
since it's just a news spool, reliability in face of a system crashes
really don't matter all that much.  On the other hand, INN's Cyclic
News File System running on top of ext2 is probably an even better
choice in that case.  What do you want to use?

What I want to get at is that you can troll the mailing lists (and
crossposting stupid inflammatory material with an inane subject to a
bunch of mailing lists the way you have done definitely is trolling)
trying to say that whatever you're trying to sell is the best, but at
the end, if a file system is better or not is a lot more complex than
quoting just one benchmark (which, once again, is meaningless,
compressing a lot of zeroes is simple and really does not tell you
anything about real world usages).  And there are other considerations
too, even if Reiser4 would be the best thing since sliced breadd, can
I trust Hans Reiser to support Reiser4 for the next five years?  Or
will he drop support for Reiser4 the same way he dropped support for
the old Reiser3 when Reiser4 came along?  Or will he drop Reiser4 when
the grant to do Reiser 4 development expires?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: Reiser4. BEST FILESYSTEM EVER - Christer Weinigel

[johnrobertbanks]

Christer Weinigel: Until YOU, have actually used the REISER4 filesystem
yourself, I think YOU OWE IT to the people on the linux-kernel mailing
list, to, AS YOU SAY, shut the fuck up. 

Even reading up on the REISER4 filesystem would help. 

Applying a little intelligence would undoubtedly help too.

> johnrobertbanks@fastmail.fm writes:
> 
> > Lennart. Tell me again that these results from 
> > 
> > http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> > http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> > 
> > are not of interest to you. I still don't understand why you
> > have your head in the sand.
> 
> Oh, for fucks sake, stop sounding like a broken record.  

Oh, for fucks sake, would you, and your religious anti-REISER cohorts,
stop sounding like a broken record.

> You have repeated the same totally meaningless statistics more 
> times than I care to count.  Please shut the fuck up.

You, and your religious anti-REISER cohorts, have indeed repeated the
same broken arguments (if you can call them such) more times than I care
to count.

NO statistics, NO real facts, just selective MANIPULATION of facts.

> Please shut the fuck up.

Yes, why don't you politely, shut the fuck up.

Until YOU, have actually used the REISER4 filesystem yourself, I think
YOU OWE IT to the people on the linux-kernel mailing list, to shut the
fuck up, as YOU say.

I guess, the fact that you are a TOTAL HYPOCRITE, has completely escaped
you.

By the way: Did I thank you "delightful" people for the "pleasant"
welcome to the linux-kernel mailing list?

-----------------------------------------------------------------

> So the two bonnie benchmarks with lzo and gzip are
> totally meaningless for any real life usages.

YOU (yes, the one with no experience and next to NO knowledge on the
subject) claim that because bonnie++ writes files that are mostly zeros,
the results are meaningless. It should be mentioned that bonnie++ writes
files that are mostly zero for all the filesystems compared. So the
results are meaningful, contrary to would you claim.

And hopefully all will notice that you just ignore these tests:

.-------------------------------------------------.
|File         |Disk |Copy |Copy |Tar  |Unzip| Del |
|System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
|Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
.-------------------------------------------------.
|REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
|REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
|REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
|REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
|NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
|NTFS         | 779 | 781 | 173 |   X |   X |   X |
|REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
|XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
|JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
|EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
|EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
|EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
|EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
|FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
.-------------------------------------------------.


where the files are definitely NOT mostly zeros. 

Your negligence has to be deliberate,... but why?

Are you manipulating the facts just to try and win an argument?

Most sane people will realize, that what you say is simply wrong.

ALSO YOU IGNORE examples offered by others, on lkml, which contradict
your assertion: FOR EXAMPLE:

> I see the same thing with my nightly scripts that do syslog analysis, last year 
> I trimmed 2 hours from the nightly run by processing compressed files instead of 
> uncompressed ones (after I did this I configured it to compress the files as 
> they are rolled, but rolling every 5 min the compression takes <20 seconds, so 
> the compression is < 30 min)

>From David Lang http://lkml.org/lkml/2007/4/7/196

Willy Tarreau also mentions this situation in a couple of articles.

Let me spoon feed you:

David has said that compressing the logs takes

24 x 12 x 20 secs = 5,760 secs = 1.6 hours of CPU time (over the day)

but he saves 2 hours of CPU time on the daily syslog analysis.

For a total (minimum) saving of 24 minutes.

The actual saving is probably much greater. It depends on the CPU
utilization when not compressing, ie, whether you are using ide CPU
cycles or not. I guess it also depends on whether you can go home one
and a half hours earlier by using compression, or if your boss makes you
stick around anyway.

NOTE THAT THE FILES IN THIS EXAMPLE ARE ALSO NOT MAINLY ZEROS.

MAYBE you just lacked the knowledge to understand what David was saying,
or maybe your desire to denigrate REISER4 is so strong, that you simply
don't care what other people say about similar circumstances.

I am not sure why you have to be spoon feed on these matters, or why you
adamantly refuse to find the facts of the matter, for yourself.

-----------------------------------------------------------------
-----------------------------------------------------------------

On Fri, 6 Apr 2007 23:30:49 -0400, "Jan Harkes" <jaharkes@cs.cmu.edu>
said:
> 
> Since you decide to publically respond to a private email, but not only
> you did not 'discuss' anything I wrote and in fact cut out most of the
> useful information in my reply I guess I will have to repeat my
> observations.

You are a funny guy Jan.

Here you are, once again, cutting out my most useful information, ie,
the data I was discussing, while complaining that I cut out your most
useful information. 

You know,... you cut out this bit:

-----------------------------------------------------------------

> The following benchmarks are from
> 
> http://linuxhelp.150m.com/resources/fs-benchmarks.htm or,
> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> .-------------------------.
> | FILESYSTEM | TIME |DISK |
> | TYPE       |(secs)|USAGE|
> .-------------------------.
> |REISER4 lzo | 1938 | 278 |
> |REISER4 gzip| 2295 | 213 |
> |REISER4     | 3462 | 692 |
> |EXT2        | 4092 | 816 |
> |JFS         | 4225 | 806 |
> |EXT4        | 4408 | 816 |
> |EXT3        | 4421 | 816 |
> |XFS         | 4625 | 779 |
> |REISER3     | 6178 | 793 |
> |FAT32       |12342 | 988 |
> |NTFS-3g     |10414 | 772 |
> .-------------------------.
> 
> Column one measures the time taken to complete the bonnie++ benchmarking test (run with the parameters bonnie++ -n128:128k:0)
> 
> Column two, Disk Usage: measures the amount of disk used to store 655MB of raw data (which was 3 different copies of the Linux kernel sources).

-----------------------------------------------------------------

And this bit: 

Jan,... Here is another section, you conveniently cut out. Maybe you
should explain why you cut out this section? Was it embarrassing to you?
I mean, your statement is sort of correct,... but it shows a basic
misunderstanding of filesystems.

> > With compression there is a pretty high probability that one corrupted
> > byte or disk block will result in loss of a considerably larger amount
> > of data. 
> 
> Bad blocks are NOT dealt with by the filesystem,... so your comment is
> irrelevant, or just plain wrong.
> 
> If your filesystem is writing to bad blocks, then throw away your
> operating system.

-----------------------------------------------------------------

It is true that I considered your "most useful information," an
irrelevant section, which is why it was cut out (ignored).

I did not see my doing this, any worse than you doing it. I did not
realize that you were being so impolite.

As to your email being private, I had thought I had joined a mailing
list. I had no idea your email was meant to be private and just
considered it like all the others.

Now you mention it, I wondered why the email did not automatically list
the mailing lists, as recipients, and I had to add them. If I had
realized this I may have added the mailing lists as recipients, anyway.
It would be like me to do such. However, you should understand that I am
new to mailing lists.
 
> 
> > > Do you really have to repeat the results in every email you sent?
> > 
> > Damn, I did it again. WHY DO YOU CARE?
> 
> Because I saw them the first time around. And although the performance
> difference on those micro benchmarks seems quite impressive, I'm not
> convinced.

So, likewise, I saw your comments (you know the ones you miss so much)
the first time around, and I ignored them, as I was not convinced of
their worth.

The benchmarks measure certain data. Its fine you do not read into them,
stuff that isn't there, like reliability, for example. 

> > Look, its simple, I am (among other things) discussing these results, so
> > people need to see them.
> 
> However, you do not discuss, you just repeat, and repeat, and repeat.

I never said I wanted discussion, you just said I did.

You just repeat, and repeat, and repeat.

In reality, I quite appreciate reasonable discussion. But, I doubt I
will get much from you.

> But for what reason. Do you want an actual discussion, or do you hate
> the reiserfs developers so much that you want to antagonize any and all
> other Linux file systems developers?

Why do you think I hate reiserfs developers? That is an insane claim.
Why would I hate reiser3 developers?
Why would I hate reiser4 developers? 
Why would I even dislike them?

I think Hans Reiser is a genius. Is that what you mean by hate?

Answer this question. Why do YOU think I am antagonizing reiserfs
developers?

You must have a reason for stating what you have.

> > By the way, I have pulled the plug on my REISER4 system, a number of
> > times now, and it recovers without problem.
> 
> Very nice for you. I guess you have also not been hit by out of memory
> conditions or failing partial writes. So I guess we can ignore the patch
> that was just sent a day or two ago to the mailing list because you
> succesfully pulled the plug, a number of times at that.

Why are you attacking me with sarcasm, when I have just stated a simple
fact?

> > > > I have set up a Reiser4 partition with gzip compression, here is the
> > > > difference in disk usage of a typical Debian installation on two 10GB
> > > > partitions, one with Reiser3 and the other with Reiser4.
> > > > 
> > > > debian:/# df
> > > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > > /dev/sda3             10490104   6379164   4110940  61% /3
> > > > /dev/sda7              9967960   2632488   7335472  27% /7
> > > ...
> > > > Partition 3 is Reiser3 -- uses 6.4 GB.
> > > > Partition 7 is Reiser4 -- uses 2.6 GB.
> > > > 
> > > > So Reiser4 uses 2.6 GB to store the (typical) data that it takes Reiser3
> > > > 6.4 GB to store (note it would take ext2/3/4 some 7 GB to store the same
> > > > info).
> > > 
> > > Wow, consider me totally and completely, unimpressed.
> > > 
> 
> Here is the part of my email that you seemed to totally ignore,

Yes, I ignored it,.... is that a crime?

> You've just saved yourself $3.80, now go get yourself a latte.
> (see. http://tomayko.com/weblog/2006/09/11/that-dilbert-cartoon)
> 
> Seriously, disk storage is getting less expensive all the time, you can
> already buy a 250GB SATA drive for $70. Also, compression doesn't help
> once you store already compressed data such as jpeg images, mp3 files,
> or mpeg2/4/divx video files. Not only are the savings non-existant, but
> we still end up with the disadvantage that corruption propagates to
> multiple files because of the compression in the file system.

I believe that compression in REISER4 is just a wrapper for gzip or lzo.
So each file is compressed and your "corruption propagation" is just in
your imagination.

> And if it doesn't propagate across multiple files, the compression can't
> be all that good 

It can be as good as gzipping every file, ie, pretty damn good.

> since it can't benefit as much from similarity between
> files. So if that is the case and you really want to save diskspace you
> almost have to look at read-only compressed filesystems such as cramfs,
> squashfs, zisofs, cloop and various other variants in combination with
> a unionfs overlay to get read/write functionality.
> 
> But in the end everything is a tradeoff. You can save diskspace, but
> increase the cost of corruption. 

You deliberately ignored the fact that bad blocks are NOT dealt with by
the filesystem,... but by the operating system. Like I said: If your
filesystem is writing to bad blocks, then throw away your operating
system.

> Use a better compression algorithm, but
> that uses more CPU or which is visible in performance of the
> application. 

Not necessarily, just look at the bonnie++ tests. The tests show that
using compression (with todays hardware) leads to read speeds that are
some FOUR times the physical disk read rate,... 

but you ignore this as you have some strange anti-REISER religion.

Think about it,... read speeds that are some FOUR times the physical
disk read rate,... impossible without the use of compression (or
something similar).

> This can be offset by caching more, and being lazier with
> writebacks, but that hurts on-disk consistency, creates more memory
> pressure (more swapout/paging) and generally slows down other
> applications that aren't actually accessing the disk. Having a fast
> multi-core cpu and lots of memory helps a lot, but at some point what
> tradeoff did we just make, we saved a couple of dollars not having to
> buy a larger disk, but we spend considerably more on the more
> expensive cpu and memory.
> 

Wow, consider me STILL totally impressed by your AMAZING BIAS.
Would you like to tell me why you are SO BIASED against REISER4?

John.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - The professional email service

Re: Reiser4. BEST FILESYSTEM EVER - Christer Weinigel

[Richard Knutsson]

Wow, I'm impressed. Think you got the record on how many mails you 
referenced to in a reply... But dude, please calm down, the caps-lock is 
not the answer. You have got some rude answers and you have called them 
back on it + you have repeated the same statement several times, that is 
not the best way of convincing people.

I believe you picked up the "anti-Reiser religion"-phrase from previous 
rant-wars (otherwise, why does that "religion"-phrase always come up, 
and (almost) only when dealing with Reiser-fs), and yes, there has been 
some clashes caused by both sides, so please be careful when dealing 
with this matter.

Would you be willing to benchmark Reiser4 with some compressed 
binary-blob and show the time as well as the CPU-usage? And document how 
it is set up so it can be reproduced. After all, Windows is suppose to 
be more stable, maintained and cost-efficient then Linux, but they don't 
tell us how ;)

>> since it can't benefit as much from similarity between
>> files. So if that is the case and you really want to save diskspace you
>> almost have to look at read-only compressed filesystems such as cramfs,
>> squashfs, zisofs, cloop and various other variants in combination with
>> a unionfs overlay to get read/write functionality.
>>
>> But in the end everything is a tradeoff. You can save diskspace, but
>> increase the cost of corruption. 
>>     
>
> You deliberately ignored the fact that bad blocks are NOT dealt with by
> the filesystem,... but by the operating system. Like I said: If your
> filesystem is writing to bad blocks, then throw away your operating
> system.
>   
I may have missed something, but if my room-mate took my harddrive, 
screwed it open, wrote a love-letter on the disk with a pencil and then 
returned it (ok, there may be some more plausible reasons for 
corruption), is the OS really suppose to handle it? Yes, it should not 
assign any new data to those blocks but should it not also fall into the 
file-systems domain to be able to restore some/all data?


Just my 2c to the pond
Richard Knutsson

Re: Reiser4. BEST FILESYSTEM EVER - Christer Weinigel

[johnrobertbanks]

On Mon, 09 Apr 2007 00:58:53 +0200, "Richard Knutsson"
<ricknu-0@student.ltu.se> said:
> Wow, I'm impressed. Think you got the record on how many mails you 
> referenced to in a reply... 

TWO actually. I guess you are easily impressed.

A simple cut and paste error.

> You have got some rude answers and you have called them back on it 

Yeah, I (fairly closely) mimicked their behavior to make a point.

> + you have repeated the same statement several times, that is 
> not the best way of convincing people.

I know you DON'T believe that, as you are about the tenth person to
repeat that "repeating stuff has no effect."

> I believe you picked up the "anti-Reiser religion"-phrase from previous 
> rant-wars (otherwise, why does that "religion"-phrase always come up, 
> and (almost) only when dealing with Reiser-fs), and yes, there has been 
> some clashes caused by both sides, so please be careful when dealing 
> with this matter.

NO. You people simply come across as zealots who work together, against
Reiser4.

Hence the term "anti-Reiser religion."

> Would you be willing to benchmark Reiser4 with some compressed 
> binary-blob and show the time as well as the CPU-usage? 

I might be. I don't really know how to set it all up.

Perhaps if you guided me through it.

> >
> > You deliberately ignored the fact that bad blocks are NOT dealt with by
> > the filesystem,... but by the operating system. Like I said: If your
> > filesystem is writing to bad blocks, then throw away your operating
> > system.
> >   

> I may have missed something, but if my room-mate took my harddrive, 
> screwed it open, wrote a love-letter on the disk with a pencil and then 
> returned it (ok, there may be some more plausible reasons for 
> corruption), is the OS really suppose to handle it? 

Yeah, I can't see how the OS could read the love-letter either.

But one thing is for sure. The FS ain't responsible for reading it.

> Yes, it should not 
> assign any new data to those blocks but should it not also fall into the 
> file-systems domain to be able to restore some/all data?

It's a tough ask of any FS. 

Microsoft's filesystem checker totally roasted all my data on an XP-box
last night. 

I had used ntfsresize to reduce the partition size and had a power
outage. 

Later, Windows booted, ran the filesystem checker, seemed OK. 

Next time I boot, all I get is Input/Output error.

> 
> Just my 2c to the pond
> Richard Knutsson
> 
Addin my 2c
John.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - A no graphics, no pop-ups email service

Re: Reiser4. BEST FILESYSTEM EVER - Christer Weinigel

[Willy Tarreau]

On Sun, Apr 08, 2007 at 10:14:18PM -0700, johnrobertbanks@fastmail.fm wrote:
> NO. You people simply come across as zealots who work together, against
> Reiser4.

Poor guy ! People are not against Reiser4, they are against the stupid and
irritating person who pollutes the lists always sending the same results
without any comment because he doesn't even understand the results. Just
like the kid on the beach "Look Ma, I found a soft shell!". "Leave it
overthere, it's a jellyfish !".  "I don't know what a jellyfish is, I will
take all those soft shells with me".

You keep saying people do not want to read you, but there is nothing to read.
In fact, you hope that people will comment on your results so that you will
finally understand them, but people keep saying there is nothing to read there.

When will you stop annoying people with your noisy toys ?

Oh, and please send your conspiracy claims somewhere else.

Willy

Re: Reiser4. BEST FILESYSTEM EVER - Christer Weinigel

[Richard Knutsson]

johnrobertbanks@fastmail.fm wrote:
> On Mon, 09 Apr 2007 00:58:53 +0200, "Richard Knutsson"
> <ricknu-0@student.ltu.se> said:
>   
>> Wow, I'm impressed. Think you got the record on how many mails you 
>> referenced to in a reply... 
>>     
>
> TWO actually. I guess you are easily impressed.
>   
Oh, took it to be from 5-6 sources...
>> + you have repeated the same statement several times, that is 
>> not the best way of convincing people.
>>     
>
> I know you DON'T believe that, as you are about the tenth person to
> repeat that "repeating stuff has no effect."
>   
Why should we change our response to the same error? The only solution 
to this loop is when people stops answering you and you "lose".
>> I believe you picked up the "anti-Reiser religion"-phrase from previous 
>> rant-wars (otherwise, why does that "religion"-phrase always come up, 
>> and (almost) only when dealing with Reiser-fs), and yes, there has been 
>> some clashes caused by both sides, so please be careful when dealing 
>> with this matter.
>>     
>
> NO. You people simply come across as zealots who work together, against
> Reiser4.
>
> Hence the term "anti-Reiser religion."
>   
Please, don't address someone you meet for the first time as "you people"!
Yes, we do _work_ together, it is a community and as a community you 
have to follow the social rules agreed upon. Without all those 
pro-Reiser peoples who knew how to work with the rest, there would not 
be a ResierFS/Reiser3 in the kernel. Unfortunately, Hans is in this case 
his own worst enemy and has ruffed quite a few feathers over the time. I 
don't think you would like someone who tells you "if you do it my way, 
then you are doing it wrong"...

But personally, even if I find Hans a bit too strong-headed, he got some 
interesting design-ideas and the Reiser-filesystem is something I think 
many find interesting as a concept but not yet trust-worthy for their 
own machines.
>> Would you be willing to benchmark Reiser4 with some compressed 
>> binary-blob and show the time as well as the CPU-usage? 
>>     
>
> I might be. I don't really know how to set it all up.
>
> Perhaps if you guided me through it.
>   
Am not sure how much help I would be but from the responses to your 
benchmark-list, there seems to be many who could help you. But first I 
think you should set up a system to test on, and then after some tests 
and made the result public, there will (most likely) be people who ask 
you to test it in some specific way.
>> I may have missed something, but if my room-mate took my harddrive, 
>> screwed it open, wrote a love-letter on the disk with a pencil and then 
>> returned it (ok, there may be some more plausible reasons for 
>> corruption), is the OS really suppose to handle it? 
>>     
>
> Yeah, I can't see how the OS could read the love-letter either.
>
> But one thing is for sure. The FS ain't responsible for reading it.
>   
And no-one has asked the file-system to _read_ the disk, but to be 
designed to help restore the file-structure. This I have found to be the 
main-point people complains about.
It is like arguing against air-bags in a car. Of course the car should 
not be responsible for preventing accidents, but they are designed so 
_if_ it happens, you should not be totally screwed.
>> Yes, it should not 
>> assign any new data to those blocks but should it not also fall into the 
>> file-systems domain to be able to restore some/all data?
>>     
>
> It's a tough ask of any FS. 
>
> Microsoft's filesystem checker totally roasted all my data on an XP-box
> last night. 
>   
Sorry to hear that, but two wrongs does not make it right.

Richard Knutsson

Re: Reiser4. BEST FILESYSTEM EVER.

[Nate Diller]

On 08 Apr 2007 06:32:26 +0200, Christer Weinigel <christer@weinigel.se> wrote:
> johnrobertbanks@fastmail.fm writes:
>
> > Lennart. Tell me again that these results from
> >
> > http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> > http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> >
> > are not of interest to you. I still don't understand why you have your
> > head in the sand.
>
> Oh, for fucks sake, stop sounding like a broken record.  You have
> repeated the same totally meaningless statistics more times than I
> care to count.  Please shut the fuck up.

wow, it's really amazing how reiser4 can still inspire flamewars so
easily when Hans isn't even around to antagonize people and escalate
things

> As you discovered yourself (even though you seem to fail to understand
> the significance of your discovery), bonnie writes files that consist
> of mostly zeroes.  If your normal use cases consist of creating a
> bunch of files containing zeroes, reiser4 with compression will do
> great.  Just lovely.  Except that nobody sane would store a lot of
> files containing zeroes, except an an excercize in mental
> masturbation.  So the two bonnie benchmarks with lzo and gzip are
> totally meaningless for any real life usages.

yeah, i sure wish Grev was still around running the benchmarks and
regression testing, cause I thought she came up with a good, QA
oriented mix of real benchmarks.  aside from a number of streaming
video benchmarks i did, those were the only results i actually trusted
to compare reiser4 with other systems.  I know Ted doesn't like the
Mongo suite, cause it focuses on small files and shows the common
weakness of block-aligned storage ... personally i thought it was
great for its primary purpose, making sure reiser4 was optimized for
its target workload.  i also recall that the distribution of small
files to large ones in mongo was pulled from some paper out of CMU,
but i can't find the reference to that study right now.

> As for the amount of disk needed to store three kernel trees, the
> figures you quote show that Reiser4 does tail combining where the tail
> of multiple files are stored in one disk block.  A nice trick that
> seems save you about 15% disk space compared to ext3.  Now you have to
> realise what that means, it means that if the disk block containing
> those tails (or any metadata pointing at that block) gets corrupted,
> instead of just losing one disk block for one file, you will have lost
> the tail for all the files sharing that disk block.  Depending on your
> personal prioritites, saving 15% of the space may be worth the risk to
> you, or maybe not.  Personally, for the only disk I'm short on space
> on, I mostly store flac encoded images of my CD collection, and saving
> 2kByte out of every 300MByte disk simply doesn't make any difference,
> and I much prefer a stable file system that I can trust not to lose my
> data.  You might make different choices.

well, it turns out that reiser4 does things a little differently,
since tail packing has bad performance effects (i always turn it off
on my reiserfs partitions).  Reiser4 guarantees a file will be stored
contiguously if it is below a certain size (20K?), and instead stores
the whole file unaligned, so that many files can be packed together
without slack space.  this gives the best of both worlds
performance-wise, at the expense of some complicated flush code to
pack everything together in the tree before it gets written.  that
combined with the fine-grained locking scheme (per-node -- reiserfs
just has a global lock) is the primary reason the code is so
convoluted ... not poor coding.

> The same goes for just about every feature that you tout, it has its
> advantages, and it has its disadvantages.  Doing compression on data
> is great if the data you store is compressible, and sucks if it isn't.
> Doing compression on each disk block and then packing multiple
> compressed blocks into each physical disk block will probably save
> some space if the data is compressible, but at the same time it means
> that you will spend a lot of CPU (and cache footprint) compressing and
> uncompressing that data.  On a single user system where the CPU is
> mostly idle it might not make much of a difference, on a heavily
> loaded multiuser system it might do.

my understanding of the code is that it uses a heuristic to decide if
a file is already compressed, so that the system doesn't waste time on
them and simply writes them out directly.  there may also be a way to
turn it off for certain classes of files, this would be most useful
for executables and the like that are frequently mmap()ed and we care
more about page-alignment than read bandwidth or data density.
edward?

> Logs can be compressed quite well using a block based compression
> scheme, but the logs can be compressed even better by doing
> compression on the whole file with gzip.  So what's the best choice,
> to do transparent compression on the fly giving ok compression or
> teaching the userspace tools to do compression of old logs and get
> really good compression?  Or maybe disk space really isn't that
> important anyway and the best thing is to just leave the logs
> uncompressed.

i guess the idea with reiser4's compression (encryption and
compression, actually) is that you can get the feature for files you
care about without having to use *extra* CPU time, it only does the
work at flush time so that you can take advantage of cache effects for
files that see lots of modifications.  ATM i doubt this works well
though, cause you'd have to manually increase dirty_background_ratio
to keep things from continually flushing and using background CPU

NATE

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[johnrobertbanks]

This is a reply to an email that I accidentally deleted.

COMPILING AND CONFIGURING A NEW KERNEL.

Download a recent kernel from http://www.kernel.org/
I will use the kernel linux-2.6.20.tar.bz2

You will have to change details of the following to suit your purposes.

Save it in /usr/src/
# mv linux-2.6.20.tar.bz2 /usr/src/

Unzip the kernel package
# tar -jxf linux-2.6.20.tar.bz2

Copy the original kernel configuration file (that came with your distro)
to .config
# cp /boot/config-2.6.20 /usr/src/linux-2.6.20/.config

Look at the available kernel building options
# make help

Run oldconfig to update the original kernel config to a current config
# make oldconfig

Use menuconfig (or xconfig or gconfig) to make any further changes
# make menuconfig

YOU SHOULD compile all the drivers necessary to boot your system, into
the kernel (ie, such drivers should not be built as modules).

This way you will NOT need an initrd file.

Use rpm-pkg to create a Red Hat RPM kernel package.
# make rpm-pkg

Went built, the RPM package is put in
/usr/src/packages/RPMS/*your*architecture*

# cd /usr/src/packages/RPMS/x86_64

Install the package (you may have to un-install previous installs)
# rpm -i kernel-2.6.20-1.x86_64.rpm

Use deb-pkg to create a Debian .deb kernel package.
# make deb-pkg

Went built, the .deb package is put in /usr/src/
# cd /usr/src/

Install the package (you may have to un-install previous installs)
# dpkg --install linux-2.6.20_2.6.20_amd64.deb

If you were unable to determine which drivers you need (to boot), then
you will need an initrd file. To build it use the command
# mkinitrd -o /boot/initrd-2.6.20

IF YOU ARE CUSTOMIZING YOUR KERNEL, YOU SHOULD PUT IN THE EFFORT TO
BUILD A KERNEL THAT DOES NOT NEED AN INITRD FILE.

------------------------------------------------------

Now you need to configure your kernel. Using GRUB you need to change the
menu.lst file.

# emacs /boot/grub/menu.lst &

The grub entry that you boot with will look something like:

###Don't change this comment - YaST2 identifier: Original name: linux###
title SUSE LINUX 10.0
    root (hd0,2)
    kernel /boot/vmlinuz root=/dev/hda3 resume=/dev/hda5 vga=0x317
    video=vesafb:nomtrr splash=silent
    initrd /boot/initrd

Leave the old boot entry so you can boot it if things go wrong with the
new kernel.
Cut and paste a copy of it above the old one. Then adjust the copy for
the new kernel.

###Don't change this comment - YaST2 identifier: Original name: linux###
title MY NEW KERNEL
    root (hd0,2)
    kernel /boot/linux-2.6.20 root=/dev/hda3 resume=/dev/hda5 vga=0x317
    video=vesafb:nomtrr splash=silent

Of course, you don't need a initrd entry as you have compiled in all the
vital drivers,... right?
If you could not determine the vital drivers and needed to build an
initrd file, then you need an entry, like

    initrd /boot/initrd-2.6.20

------------------------------------------------------

If your new kernel is destined to have the same name as the old one, you
need to do something about it (unless you do not mind the old one being
overwritten).

Use your favorite text editor to change the top level Makefile
# emacs /usr/src/linux-2.6.20/Makefile &

change the line 
EXTRAVERSION
to 
EXTRAVERSION = something

This will change the name of the new kernel to linux-2.6.20-something

Your /boot/grub/menu.lst entry will now look something like:

###Don't change this comment - YaST2 identifier: Original name: linux###
title MY NEW KERNEL
    root (hd0,2)
    kernel /boot/linux-2.6.20-something root=/dev/hda3 resume=/dev/hda5
    vga=0x317 video=vesafb:nomtrr splash=silent

and perhaps an entry

    initrd /boot/initrd-2.6.20-something

------------------------------------------------------

Now reboot and choose the "MY NEW KERNEL" entry from the GRUB boot menu,
and see how you went.


------------------------------------------------------


On Thu, 05 Apr 2007 17:05:21 -0700, johnrobertbanks@fastmail.fm said:
> Hi Ignatich,
> 
> After seeing the following benchmarks at 
> 
> http://linuxhelp.150m.com/resources/fs-benchmarks.htm and
> http://m.domaindlx.com/LinuxHelp/resources/fs-benchmarks.htm
> 
> The Reiser4 benchmarks are so good, I have decided to try the Reiser4
> filesystem.
> 
> .-------------------------.
> | FILESYSTEM | TIME |DISK |
> | TYPE       |(secs)|USAGE|
> .-------------------------.
> |REISER4 lzo | 1938 | 278 |
> |REISER4 gzip| 2295 | 213 |
> |REISER4     | 3462 | 692 |
> |EXT2        | 4092 | 816 |
> |JFS         | 4225 | 806 |
> |EXT4        | 4408 | 816 |
> |EXT3        | 4421 | 816 |
> |XFS         | 4625 | 779 |
> |REISER3     | 6178 | 793 |
> |FAT32       |12342 | 988 |
> |NTFS-3g     |10414 | 772 |
> .-------------------------.
> 
> Column one measures the time taken to complete the bonnie++ benchmarking
> test (run with the parameters bonnie++ -n128:128k:0)
> 
> Column two, Disk Usage: measures the amount of disk used to store 655MB
> of raw data (which was 3 different copies of the Linux kernel sources).
> 
> Anyway, I have patched the 2.6.20 kernel and have a partition formatted
> with Reiser4.
> 
> However, I am having trouble getting LILO or GRUB working (with
> Reiser4).
> 
> Could you guys who know all about this, help me, or point me to some
> help.
> 
> Thanks a lot, John.
> 
> 
> On Fri, 06 Apr 2007 02:42:35 +0400, "Ignatich" <ignatich@gmail.com>
> said:
> > While trying to find the cause of problems with reiser4 in recent 
> > kernels I came across this.
> > 
> > Incomplete write handling seem to be missing from reiser4_write_extent() 
> > thanks to reiser4-temp-fix.patch. Strangely, there is a patch by Edward 
> > Shishkin that should address that issue, but it is missing from -mm 
> > tree. Please check.
> > 
> >     Max
> > 
> -- 
>   
>   johnrobertbanks@fastmail.fm
> 
> -- 
> http://www.fastmail.fm - And now for something completely different…
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - mmm... Fastmail...

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[johnrobertbanks]

Just correcting some errors and typos. 

Wouldn't want you to say that the linux kernel mailing list gave you
incorrect info.

COMPILING AND CONFIGURING A NEW KERNEL.

Download a recent kernel from http://www.kernel.org/
I will use the kernel linux-2.6.20.tar.bz2

You will have to change details of the following to suit your purposes.

Save it in /usr/src/
# mv linux-2.6.20.tar.bz2 /usr/src/

Unzip the kernel package
# tar -jxf linux-2.6.20.tar.bz2

Copy the original kernel configuration file (that came with your distro)
to .config
# cp /boot/config-2.6.13-15-default /usr/src/linux-2.6.20/.config

Change to the new kernel source directory
# cd /usr/src/linux-2.6.20/

Look at the available kernel building options
# make help

Run oldconfig to update the original kernel configuration to a current
configuration
# make oldconfig

Use menuconfig (or xconfig or gconfig) to make any further changes
# make menuconfig

YOU SHOULD compile all the drivers necessary to boot your system, into
the kernel (ie, such drivers should not be built as modules).

This way you will NOT need an initrd file.

Use rpm-pkg to create a Red Hat RPM kernel package.
# make rpm-pkg

When built, the RPM package is put in
/usr/src/packages/RPMS/*your*architecture*

# cd /usr/src/packages/RPMS/x86_64

Install the package (you may have to un-install previous installs)
# rpm -i kernel-2.6.20-1.x86_64.rpm

Use deb-pkg to create a Debian .deb kernel package.
# make deb-pkg

When built, the .deb package is put in /usr/src/
# cd /usr/src/

Install the package (you may have to un-install previous installs)
# dpkg --install linux-2.6.20_2.6.20_amd64.deb

If you were unable to determine which drivers you need (to boot), then
you will need an initrd file. To build it use the command
# mkinitrd -o /boot/initrd-2.6.20

IF YOU ARE CUSTOMIZING YOUR KERNEL, YOU SHOULD PUT IN THE EFFORT TO
BUILD A KERNEL THAT DOES NOT NEED AN INITRD FILE.

It is possible that deb-pkg and rpm-pkg take care of creating the initrd
automatically.

I have always compiled in the important drivers, so I do not know.

Does any caring person here know the answer to this question?

------------------------------------------------------

Now you need to configure your kernel. Using GRUB you need to change the
menu.lst file.

# emacs /boot/grub/menu.lst &

The grub entry that you presently boot with, will look something like:

###Don't change this comment - YaST2 identifier: Original name: linux###
title SUSE LINUX 10.0
    root (hd0,2)
    kernel /boot/vmlinuz-2.6.13-15-default root=/dev/hda3
    resume=/dev/hda5 vga=0x317 video=vesafb:nomtrr splash=silent
    initrd /boot/initrd

Do NOT delete the old boot entry, so you can boot it, if things go wrong
with the new kernel.

Cut a copy of it and paste it above the original. Then adjust the copy
for the new kernel.

###Don't change this comment - YaST2 identifier: Original name: linux###
title MY NEW KERNEL
    root (hd0,2)
    kernel /boot/linux-2.6.20 root=/dev/hda3 resume=/dev/hda5 vga=0x317
    video=vesafb:nomtrr splash=silent

Of course, you don't need a initrd entry as you have compiled in all the
vital drivers,... right?
If you could not determine the vital drivers and needed to build an
initrd file, then you need an entry, like

    initrd /boot/initrd-2.6.20

------------------------------------------------------

If your new kernel is destined to have the same name as the old one, you
need to do something about it (unless you do not mind the old one being
overwritten).

Use your favorite text editor to change the top level Makefile
# emacs /usr/src/linux-2.6.20/Makefile &

change the line 
EXTRAVERSION
to 
EXTRAVERSION = something

This will change the name of the new kernel to linux-2.6.20-something

Your /boot/grub/menu.lst entry will now look something like:

###Don't change this comment - YaST2 identifier: Original name: linux###
title MY NEW KERNEL
    root (hd0,2)
    kernel /boot/linux-2.6.20-something root=/dev/hda3 resume=/dev/hda5
    vga=0x317 video=vesafb:nomtrr splash=silent

and perhaps an entry

    initrd /boot/initrd-2.6.20-something

------------------------------------------------------

Now reboot and choose the "MY NEW KERNEL" entry from the GRUB boot menu,
and see how you went.

-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Send your email first class

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On Sat, 07 Apr 2007 00:45:32 PDT, johnrobertbanks@fastmail.fm said:

> Use rpm-pkg to create a Red Hat RPM kernel package.
> # make rpm-pkg
> 
> When built, the RPM package is put in
> /usr/src/packages/RPMS/*your*architecture*
> 
> # cd /usr/src/packages/RPMS/x86_64
> 
> Install the package (you may have to un-install previous installs)
> # rpm -i kernel-2.6.20-1.x86_64.rpm

It is *highly* recommended that you change the kernel identifier at
least slightly, so that you can install '2.6.20-1.local' without overlaying
the vendor-supplied 2.6.20-1 kernel.  Among other things, this lets you
boot back to the equivalent code level in the vendor kernel, so you can figure
out if it's your .config file that's broken, or if you hit a bug upggrading
from 2.6.19-10 to 2.6.20-1.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[johnrobertbanks]

> It is *highly* recommended that you change the kernel identifier at
> least slightly, so that you can install '2.6.20-1.local' without
> overlaying
> the vendor-supplied 2.6.20-1 kernel.  Among other things, this lets you
> boot back to the equivalent code level in the vendor kernel, 
> so you can figure
> out if it's your .config file that's broken, or if you hit a bug
> upggrading from 2.6.19-10 to 2.6.20-1.

I agree. I think your advice is *highly* recommended.

I had this problem once after forcing an upgrade, which removed the
working kernel. I just booted the kernel and stuff, from another
partition.

John.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - The way an email service should be

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

On Fri, 06 Apr 2007 18:26:45 PDT, johnrobertbanks@fastmail.fm said:
>
> YOU SHOULD compile all the drivers necessary to boot your system, into
> the kernel (ie, such drivers should not be built as modules).
> 
> This way you will NOT need an initrd file.

It is quite possible to build a kernel that has all the drivers built-in,
but still require an initrd file.  For instance, if you have a recent
RedHat or Fedora system, '/' may very well be on an LVM partition, which
means you need an initrd to do a 'lvm varyonvg' before mounting your real
root filesystem will work....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[johnrobertbanks]

> It is quite possible to build a kernel that has all the drivers built-in,
> but still require an initrd file.  For instance, if you have a recent
> RedHat or Fedora system, '/' may very well be on an LVM partition, which
> means you need an initrd to do a 'lvm varyonvg' before mounting your real
> root filesystem will work....

Thanks Valdis,

Thats interesting, I didn't know that. 

Do you know if deb-pkg and rpm-pkg take care of creating the initrd
automatically.

I seriously doubt they do.

Actually, I guess I only need to compile another kernel to find out.

John.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Choose from over 50 domains or use your own

Re: COMPILING AND CONFIGURING A NEW KERNEL.

[Lennart Sorensen]

On Sat, Apr 07, 2007 at 06:02:37PM -0700, johnrobertbanks@fastmail.fm wrote:
> Thats interesting, I didn't know that. 
> 
> Do you know if deb-pkg and rpm-pkg take care of creating the initrd
> automatically.
> 
> I seriously doubt they do.
> 
> Actually, I guess I only need to compile another kernel to find out.

Certainly on debian make-kpkg actually does have options for initrd
(--initrd) and will call mkinitrd/update-initramfs/yaird at package
install time to generate an initrd for the kernel to deal with raid,
lvm, etc.  It works quite well in general.

I would be surprised if redhat didn't do something similar.

--
Len Sorensen

Re: REISER4: fix for reiser4_write_extent

[Laurent Riffard]

Le 06.04.2007 00:42, Ignatich a écrit :
> While trying to find the cause of problems with reiser4 in recent 
> kernels I came across this.
> 
> Incomplete write handling seem to be missing from reiser4_write_extent() 
> thanks to reiser4-temp-fix.patch. Strangely, there is a patch by Edward 
> Shishkin that should address that issue, but it is missing from -mm 
> tree. Please check.
> 
>    Max
> 

This patch was added to -mm tree the 14 Dec 2006 (see 
http://www.mail-archive.com/mm-commits@vger.kernel.org/msg05338.html).

It was then dropped from -mm tree the 05 Mar 2007 (see 
http://www.mail-archive.com/mm-commits@vger.kernel.org/msg10818.html), 
with this comment:
	"This patch was dropped because it is obsolete"

No idea why it was obsolete. Does somebody know ?

~~
laurent

Re: REISER4: fix for reiser4_write_extent

[Edward Shishkin]

Laurent Riffard wrote:

> Le 06.04.2007 00:42, Ignatich a écrit :
>
>> While trying to find the cause of problems with reiser4 in recent 
>> kernels I came across this.
>>
>> Incomplete write handling seem to be missing from 
>> reiser4_write_extent() thanks to reiser4-temp-fix.patch. Strangely, 
>> there is a patch by Edward Shishkin that should address that issue, 
>> but it is missing from -mm tree. Please check.
>>
>>    Max
>>
>
> This patch was added to -mm tree the 14 Dec 2006 (see 
> http://www.mail-archive.com/mm-commits@vger.kernel.org/msg05338.html).
>
> It was then dropped from -mm tree the 05 Mar 2007 (see 
> http://www.mail-archive.com/mm-commits@vger.kernel.org/msg10818.html), 
> with this comment:
>     "This patch was dropped because it is obsolete"
>
> No idea why it was obsolete. Does somebody know ?


This uses not settled interface filemap_copy_from_user_atomic/nonatomic
However, those things should be fixed. I'll prepare the patch a bit later..

Thanks,
Edward.

Re: Reiser4. BEST FILESYSTEM EVER.

[Dale Amon]

Jan does have a point about bad blocks. A couple years ago
I had a relatively new disk start to go bad on random blocks.
I detected it fairly quickly but did have some data loss.

All the compressed archives which were hit were near
total losses; most other files were at least partially
recoverable.

It is not a matter of your operating system writing
to bad blocks. It is a matter of what happens when the
blocks on which your data sit go bad underneath you.

This issue has also been discussed by people working
with revision control system. If you are archiving
data, how do you know you if your data is still good
unless you actually need it? If you do not know it
is bad, you may well get rid of good copies thinking
you do not need the extras... it does happen.

I would be quite hesitant to go with on disk compression
unless damage was limited to only the bad bits or blocks
and did not propagate through the rest of the file.

Perhaps if everyone used hardware RAID and the RAID
automatically detected a difference due to trashed
data on one disk and flagged the admin with a warning...

BTW: I'm a CMU Alum, so who are you working with Jan?

Re: Reiser4. BEST FILESYSTEM EVER.

[johnrobertbanks]

On Sat, 7 Apr 2007 13:59:14 +0100, "Dale Amon" <amon@vnl.com> said:
> Jan does have a point about bad blocks. A couple years ago
> I had a relatively new disk start to go bad on random blocks.
> I detected it fairly quickly but did have some data loss.
> 
> All the compressed archives which were hit were near
> total losses; most other files were at least partially
> recoverable.

As you know, there is not substitute for backups. What if the disk had
totally crashed and scratched GBs of your data.

And did you ever trust those (non-compressed) executables that you saved
after recovering them from corruption?

Of course not. No one would. The fact that they were not compressed did
not save them.

You are really arguing for backups, not for one filesystem or another.

Besides, Jan claimed that corruption due to bad blocks propagates to
MULTIPLE files because of the compression in the file system. You are
arguing something different.

> It is not a matter of your operating system writing
> to bad blocks. It is a matter of what happens when the
> blocks on which your data sit go bad underneath you.
>
> This issue has also been discussed by people working
> with revision control system. If you are archiving
> data, how do you know you if your data is still good
> unless you actually need it? If you do not know it
> is bad, you may well get rid of good copies thinking
> you do not need the extras... it does happen.
> 
> I would be quite hesitant to go with on disk compression
> unless damage was limited to only the bad bits or blocks
> and did not propagate through the rest of the file.

You don't really mean that. Most backup uses compression (which
propagates errors through the rest of the file).

> Perhaps if everyone used hardware RAID and the RAID
> automatically detected a difference due to trashed
> data on one disk and flagged the admin with a warning...
> 
> BTW: I'm a CMU Alum, so who are you working with John?

I retired quite young.
-- 
  
  johnrobertbanks@fastmail.fm

-- 
http://www.fastmail.fm - Send your email first class