Re: reiser4 status (correction)

[David Masover <ninja@slaphack.com>]

Mike Benoit wrote:
> On Fri, 2006-07-21 at 16:06 -0500, David Masover wrote:
>> Mike Benoit wrote:
>>
>>> Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
>>> concerned won't it? Right now Reiser4 looks excellent on the benchmarks
>>> that don't use fsync often (mongo?), but last I recall the fsync
>>> performance was so poor it overshadows the rest of the performance. It
>>> would also probably be more useful to a much wider audience, especially
>>> if Namesys decides to charge for the repacker.
>> If Namesys does decide to charge for the repacker, I'll have to consider 
>> whether it's worth it to pay for it or to use XFS instead.  Reiser4 
>> tends to become much more fragmented than most other Linux FSes -- 
>> purely subjective, but probably true.
>>
> 
> I would like to see some actual data on this. I haven't used Reiser4 for
> over a year, and when I did it was only to benchmark it. But Reiser4
> allocates on flush, so in theory this should decrease fragmentation, not
> increase it. Due to this I question what you are _really_ seeing, or if
> perhaps it is a bug in the allocator? Why would XFS or any other
> multi-purpose file system resist fragmentation noticeably more then
> Reiser4 does.

Maybe not XFS, but in any case, Reiser4 fragments more because of how 
its journaling works.  It's the wandering logs.

Basically, when most Linux filesystems allocate space, they do try to 
allocate it contiguously, and it generally stays in the same place. 
With ext3, if you write to the middle of a file, or overwrite the entire 
file, you're generally going to see your writes be written once to the 
journal, and then again to the same place the file originally was.

Similarly, if you delete and then create a bunch of small files, you're 
generally going to see the new files created in the same place the old 
files were.

With Reiser4, wandering logs means that rather than write to the 
journal, if you write to the middle of the file, it writes that chunk to 
somewhere else on the disk, and somehow gets it down to one atomic 
operation where it simply changes the file to point to the new location 
on disk.  Which means if you have a filesystem that is physically laid 
out on disk like this (for simplicity, assume it only has a single file):

# is data
* is also data
- is free space

######*****########--------------

When you try to write in the middle (the '*' chars) -- let's say we're 
changing them to '%' chars, this happens:

######*****########%%%%%---------

Once that's done, the file is updated so that the middle of it points to 
the fragment in the new location, and the old location is freed:

######-----########%%%%%---------



Keep in mind, because of lazy writes, it's much more likely for the 
whole change to happen at once.  Here's another example:

#####------------

Let's say we just want to overwrite the file with another one of the 
same length:

#####%%%%%-------

then, commit the transaction:

-----%%%%%-------

You see the problem?  You've now split the free space in half. 
Realistically, of course, it wouldn't be by halves, but you're basically 
inserting random air holes all over the place, and your FS is becoming 
more like foam, taking up more of the free space, until you can no 
longer use the free space....  In the above example, if we then have to 
come write some huge file, it looks like this:

*****%%%%%*******

Split right in half.  Now imagine this effect multiplied by hundreds or 
thousands of files, over time...

This is why Reiser4 needs a repacker.  While it's fine for larger files 
-- I believe after a certain point, it will write twice, so looking at 
our first example:


######*****########--------------

Write to a new, temporary place:

######*****########%%%%%---------

Write back to the original place:

######%%%%%########%%%%%---------

Complete the transaction and free the temporary space:

######%%%%%########--------------


This technique is what other journaling filesystems use, and it also 
means that writing is literally twice as slow as on a non-journaling 
filesystem, or on one with wandering logs like Reiser4.  But, it's a 
practical necessity when you're dealing with some 300 gig MySQL database 
of which only small 10k chunks are changing.  Taking twice as long on a 
10k chunk won't kill anyone, but fragmenting your 300 gig database (on a 
320 gig partition) will kill your performance, and will be very 
difficult to defragment.

But on smaller files, it would be very beneficial if we could allow the 
FS to slowly fragment (to foam-ify, if you will) and defrag once a week. 
  The amount of speed gained in each write -- and read, if it's not 
getting too awful during that week -- definitely makes up for having to 
spend an hour or so defragmenting, especially if the FS can be online at 
the time.

And you can probably figure out an optimal time to wait before 
defragmenting, since your biggest fragmentation problems happen when the 
chunk of contiguous space at the end of the disk disappears, and all of 
your free space is scattered (fragmented) throughout the disk.

Anyway, that's why.  If you disable the wandering log behavior, your 
write performance drops in half.  If you don't have a repacker, your FS 
becomes very fragmented, very fast.

I apologize for my poor ASCII art, especially if I'm dead wrong...


> No Linux file system that I'm aware of has a defragmentor, but they DO
> become fragmented, just not near as bad as FAT32 used to when MS created
> their defragmentor. The highest "non-contiguous" percent I've seen with
> EXT3 is about 12%, FAT32 I have seen over 50%, and NTFS over 30%. In

I'd like to see some numbers on Reiser4, then.  Maybe a formal 
fragmentation benchmark?