reiser4 status (correction)

reiser4 status (correction)

[Hans Reiser]

Well, it seems we still aren't quite as stable as we were 6 months ago 
(the new reduced cpu usage code was extensive, as was the VFS change 
code), and we know of a bug we can reproduce using our standard tests.  
Also, it seems we can oops when a particular program is run to consume 
all memory (thanks Jate for finding it).  Hopefully things will be more 
stable next week....  Us developers are using the new code on our 
workstations without problem though.

On a more positive note, Reiser4.1 is getting closer to release....  It 
is working fine for the developer coding it, and we are scheduling code 
reviews for it and defining migration paths, etc.  Hopefully in 2 months 
it will ship.

The big issue with 4.1 is that we are having to deal with all the issues 
of REALLY allowing users to change default plugins, etc., and finding we 
missed details.  We will say more later.

Thanks for your patience,

Hans

Re: reiser4 status (correction)

[David Masover]

Hans Reiser wrote:

> On a more positive note, Reiser4.1 is getting closer to release....

Good news!  But it's been awhile since I've followed development, and 
the homepage seems out of date (as usual).  Where can I find a list of 
changes since v4?

By "out of date", I mean things like this:

"Reiser4.1 will modify the repacker to insert controlled "air holes", as 
it is well known that insertion efficiency is harmed by overly tight 
packing."

Am I to understand that this text hasn't changed since the repacker was 
essentially canceled/removed/commercialized?  Or is there actually a 
repacker somewhere usable, today, that is still scheduled to insert air 
holes by v4.1?

Removed LKML from CC for now, for no particular reason.  Add them back 
in, if you like.

Re: reiser4 status (correction)

[Hans Reiser]

David Masover wrote:

> Hans Reiser wrote:
>
>> On a more positive note, Reiser4.1 is getting closer to release....
>
>
> Good news!  But it's been awhile since I've followed development, and
> the homepage seems out of date (as usual).  Where can I find a list of
> changes since v4?
>
> By "out of date", I mean things like this:
>
> "Reiser4.1 will modify the repacker to insert controlled "air holes",
> as it is well known that insertion efficiency is harmed by overly
> tight packing."

Sigh, no, the repacker will probably be after 4.1....

The list of tasks for zam looks something like:

fix bugs that arise

debug read optimization code (CPU reduction only, has no effect on IO),
1 week est.  (would be nice if it was less)

review compression code 1 day per week until it ships.

fix fsync performance (est. 1 week of time to make post-commit writes
asynchronous, maybe 3 weeks to create fixed-reserve for write twice
blocks, and make all fsync blocks write twice)

write repacker (12 weeks).

I am not sure that putting the repacker after fsync is the right choice....

The task list for vs looks like:

* fix bugs as they arise.

* fix whatever lkml complains about that either seems reasonable, or
that akpm agrees with.

* Help edward get the compression plugins out the door.

* Improve fsck's time performance.

* Fix any V3 bugs that Chris and Jeff don't fix for us.  Which reminds
me, I need to check on whether the 90% full bug got fixed....

Re: reiser4 status (correction)

[Sarath Menon]

> Sigh, no, the repacker will probably be after 4.1....
> The list of tasks for zam looks something like:
> fix bugs that arise
> debug read optimization code (CPU reduction only, has no effect on IO),
> 1 week est.  (would be nice if it was less)
> review compression code 1 day per week until it ships.
> fix fsync performance (est. 1 week of time to make post-commit writes
> asynchronous, maybe 3 weeks to create fixed-reserve for write twice
> blocks, and make all fsync blocks write twice)
> write repacker (12 weeks).

Well, this is free software, and is not backed u with a million dollars of 
funding. Trust me, these guys are doing a great job with the options they 
have. 

This is not a fliambait, but I personally feel that this is way too great 
considering the resources that the entire team has. I am more than willing to 
wait even though I have had occasional bad experiences with reiser{3,4}. 

The bottemline is that all versions of reiserfs have had great performance, 
even though with its own hiccups (which fs doesn't have ??) and simply put, 
it is a fantastic peice of code.



-- 
My mother loved children -- she would have given anything if I had been one.
		-- Groucho Marx

Re: reiser4 status (correction)

[David Masover]

Hans Reiser wrote:

> I am not sure that putting the repacker after fsync is the right choice....

Does the repacker use fsync?  I wouldn't expect it to.

Does fsync benefit from a properly packed FS?  Probably.

Also, while I don't expect anyone else to be so bold, there is a way 
around fsync performance:  Disable it.  Patch the kernel so that any 
fsync call from userspace gets ignored, but lie and tell them it worked. 
  Basically:

  asmlinkage long sys_fsync(unsigned int fd)
  {
-       return do_fsync(fd, 0);
+       return 0;       // do_fsync(fd, 0);
  }

In order to make this sane, you should have backups and an Uninterrupted 
Power Supply.  In the case of a loss of power, the box should notice and 
immediately sync, then either shut down or software suspend.  Any UPS 
battery should be able to handle the amount of time it takes to shut the 
system off.

Since anything mission critical should have backups and a UPS anyway, 
the only problem left is what happens if the system crashes.  But system 
crashes are something you have to plan for anyway.  Disks fail -- stuff 
happens.  RAID won't save you -- the RAID controller itself will 
eventually fail.

So suppose you're running some very critical server -- for now, chances 
are it's running some sort of database.  In this case, what you really 
want is database replication.  Have at least two servers up and running, 
and consider the transaction complete not when it hits the disk, but 
when all running servers acknowledge the transaction.  The RAM of two 
boxes should be safer than the disk of one.

What about a repacker?  The best I can do to hack around that is to 
restore the whole box from backup every now and then, but this requires 
the box to be down for awhile -- it's a repacker, but not an online one. 
  In this case, the solution would be to have the same two servers 
(replicating databases), and bring first one down, and then the other.

That would make me much more nervous than disabling fsync, though, 
because now you only have the one server running, and if it goes down... 
  And depending on the size of the data in question, this may not be 
feasable.  It seems entirely possible that in some setups like this, the 
only backup you'd be able to afford would be in the form of replication.

In my own personal case, I'd prefer the repacker to tuning fsync.  But 
arguments could be made for both.

Re: reiser4 status (correction)

[Mike Benoit]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

On Fri, 2006-07-21 at 02:44 -0600, Hans Reiser wrote:
> fix fsync performance (est. 1 week of time to make post-commit writes
> asynchronous, maybe 3 weeks to create fixed-reserve for write twice
> blocks, and make all fsync blocks write twice)
> 
> write repacker (12 weeks).
> 
> I am not sure that putting the repacker after fsync is the right choice....
> 

Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
concerned won't it? Right now Reiser4 looks excellent on the benchmarks
that don't use fsync often (mongo?), but last I recall the fsync
performance was so poor it overshadows the rest of the performance. It
would also probably be more useful to a much wider audience, especially
if Namesys decides to charge for the repacker.

ReiserV3 is used on a lot of mail and squid proxy servers that deal with
many small files, and these work loads usually call fsync often. My
guess is that ReiserV3 users are the most likely to migrate to Reiser4,
because they already know the benefits of using a "Reiser" file system.
But neglecting fsync performance will just put a sour taste in their
mouth. 

On top of that, I don't see how a repacker would help these work loads
much as the files usually have a high churn rate. Packing them would
probably be a net loss as the files would just be deleted in 24hrs and
replaced by new ones.

Very few people will (or should) disable fsync as David suggests, I
don't see that as a solution at all, even if it is temporary.

-- 
Mike Benoit <ipso@snappymail.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[David Masover]

Mike Benoit wrote:

> Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
> concerned won't it? Right now Reiser4 looks excellent on the benchmarks
> that don't use fsync often (mongo?), but last I recall the fsync
> performance was so poor it overshadows the rest of the performance. It
> would also probably be more useful to a much wider audience, especially
> if Namesys decides to charge for the repacker.

If Namesys does decide to charge for the repacker, I'll have to consider 
whether it's worth it to pay for it or to use XFS instead.  Reiser4 
tends to become much more fragmented than most other Linux FSes -- 
purely subjective, but probably true.

> ReiserV3 is used on a lot of mail and squid proxy servers that deal with
> many small files, and these work loads usually call fsync often.
[...]
> But neglecting fsync performance will just put a sour taste in their
> mouth. 

So will neglecting fragmentation, only worse.  At least fsync is slow up 
front.  Fragmentation will be slow much farther in, when the mailserver 
has already been through one painful upgrade.  Charging for the repacker 
just makes it worse.

> On top of that, I don't see how a repacker would help these work loads
> much as the files usually have a high churn rate. Packing them would
> probably be a net loss as the files would just be deleted in 24hrs and
> replaced by new ones.

Depends.  Some will, some won't.  My IMAP server does have a lot of 
churning, but there's also the logs (which stay for at least a month or 
two before they rotate out), and since it's IMAP, I do leave quite a lot 
of files alone.

v3 is also used on a lot of web servers, at least where I used to work 
-- some areas will be changing quite a lot, and some areas not at all. 
Changing a lot means fragmentation will happen, not changing at all 
means repacking will help.

These issues may be helped by partitioning, if you know how you're going 
to split things up.  But then, how do you partition in the middle of a 
squid server?  A lot of people visit the same sites every day, checking 
for news, but that means plenty of logos, scripts, and other things 
won't change -- but plenty of news articles will change every couple hours.

> Very few people will (or should) disable fsync as David suggests, I
> don't see that as a solution at all, even if it is temporary.

I guess the temporary solution is to incur a pretty big performance hit. 
  But it comes back to, which is more of a performance problem, fsync or 
fragmentation?

And I really would like to hear a good counter-argument to the one I've 
given for disabling fsync.  But even if we assume fsync must stay, do we 
have any benchmarks on fragmentation versus fsync?

But maybe it's best to stop debating, since both will be done 
eventually, right?

Re: reiser4 status (correction)

[Mike Benoit]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

On Fri, 2006-07-21 at 16:06 -0500, David Masover wrote:
> Mike Benoit wrote:
> 
> > Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
> > concerned won't it? Right now Reiser4 looks excellent on the benchmarks
> > that don't use fsync often (mongo?), but last I recall the fsync
> > performance was so poor it overshadows the rest of the performance. It
> > would also probably be more useful to a much wider audience, especially
> > if Namesys decides to charge for the repacker.
> 
> If Namesys does decide to charge for the repacker, I'll have to consider 
> whether it's worth it to pay for it or to use XFS instead.  Reiser4 
> tends to become much more fragmented than most other Linux FSes -- 
> purely subjective, but probably true.
> 

I would like to see some actual data on this. I haven't used Reiser4 for
over a year, and when I did it was only to benchmark it. But Reiser4
allocates on flush, so in theory this should decrease fragmentation, not
increase it. Due to this I question what you are _really_ seeing, or if
perhaps it is a bug in the allocator? Why would XFS or any other
multi-purpose file system resist fragmentation noticeably more then
Reiser4 does.

I don't think the repacker is designed to be a "must have" for every
Reiser4 installation. If it was, I would consider Reiser4 to be
seriously flawed. Instead I think it is simply designed to improve
certain workloads that may cause high fragmentation in hopes of keeping
I/O speeds at their peek. 

Am I correct in this assumption Hans?
 
No Linux file system that I'm aware of has a defragmentor, but they DO
become fragmented, just not near as bad as FAT32 used to when MS created
their defragmentor. The highest "non-contiguous" percent I've seen with
EXT3 is about 12%, FAT32 I have seen over 50%, and NTFS over 30%. In
fact I'm running in to a fragmentation issue with ReiserV3 right now
that Jeff is working on, but it is more of a worst case scenario issue,
not a regular occurrence issue.

For "normal" workloads I doubt you would notice much difference at all
by using a repacker, 10% maybe? Which is one of the reasons you probably
haven't seen a repacker for EXT2/3, even though I'm sure it would
improve performance for some people.

-- 
Mike Benoit <ipso@snappymail.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[Andreas Schäfer]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

On 14:37 Fri 21 Jul     , Mike Benoit wrote:
> No Linux file system that I'm aware of has a defragmentor, but they DO
> become fragmented, just not near as bad as FAT32 used to when MS created
> their defragmentor.

Forgotten ext2? ;-) Funny thing: If your ext3 got too fragmented:
convert it back to ext2, defrag and reconvert it again to ext3. All of
this can be done "in place", i.e. without moving the data to other
partitions etc.

-Andreas

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[David Masover]

Andreas Schäfer wrote:
> On 14:37 Fri 21 Jul     , Mike Benoit wrote:
>> No Linux file system that I'm aware of has a defragmentor, but they DO
>> become fragmented, just not near as bad as FAT32 used to when MS created
>> their defragmentor.
> 
> Forgotten ext2? ;-) Funny thing: If your ext3 got too fragmented:
> convert it back to ext2, defrag and reconvert it again to ext3. All of
> this can be done "in place", i.e. without moving the data to other
> partitions etc.

Well, I know an ext3 partition can be mounted, unchanged, as ext2.  Of 
course, you had to cleanly unmount as ext3 first, and make sure you 
cleanly mount/unmount as ext2 before you try to mount as ext3 again...

Question, then:  Can the ext2 defrag work on a raw ext3 partition, 
without having to convert it first?

Re: reiser4 status (correction)

[Andreas Schäfer]

[-- Attachment #1: Type: text/plain, Size: 181 bytes --]

On 17:45 Fri 21 Jul     , David Masover wrote:
> Question, then:  Can the ext2 defrag work on a raw ext3 partition, without having to convert it first?

Dunno, but I don't think so

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[Maciej Sołtysiak]

Hello Andreas,

Saturday, July 22, 2006, 1:06:54 AM, you wrote:

> On 17:45 Fri 21 Jul     , David Masover wrote:
>> Question, then:  Can the ext2 defrag work on a raw ext3 partition, without having to convert it first?

> Dunno, but I don't think so
I tried that once, back in 2002 i think. defrag seemed doing things fine, but thrashed the
filesystem very badly. The good thing is, that I was able to fsck it and recover
most of my files. However it is not something you should try.

-- 
Best regards,
Maciej

Re: reiser4 status (correction)

[David Masover]

Mike Benoit wrote:
> On Fri, 2006-07-21 at 16:06 -0500, David Masover wrote:
>> Mike Benoit wrote:
>>
>>> Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
>>> concerned won't it? Right now Reiser4 looks excellent on the benchmarks
>>> that don't use fsync often (mongo?), but last I recall the fsync
>>> performance was so poor it overshadows the rest of the performance. It
>>> would also probably be more useful to a much wider audience, especially
>>> if Namesys decides to charge for the repacker.
>> If Namesys does decide to charge for the repacker, I'll have to consider 
>> whether it's worth it to pay for it or to use XFS instead.  Reiser4 
>> tends to become much more fragmented than most other Linux FSes -- 
>> purely subjective, but probably true.
>>
> 
> I would like to see some actual data on this. I haven't used Reiser4 for
> over a year, and when I did it was only to benchmark it. But Reiser4
> allocates on flush, so in theory this should decrease fragmentation, not
> increase it. Due to this I question what you are _really_ seeing, or if
> perhaps it is a bug in the allocator? Why would XFS or any other
> multi-purpose file system resist fragmentation noticeably more then
> Reiser4 does.

Maybe not XFS, but in any case, Reiser4 fragments more because of how 
its journaling works.  It's the wandering logs.

Basically, when most Linux filesystems allocate space, they do try to 
allocate it contiguously, and it generally stays in the same place. 
With ext3, if you write to the middle of a file, or overwrite the entire 
file, you're generally going to see your writes be written once to the 
journal, and then again to the same place the file originally was.

Similarly, if you delete and then create a bunch of small files, you're 
generally going to see the new files created in the same place the old 
files were.

With Reiser4, wandering logs means that rather than write to the 
journal, if you write to the middle of the file, it writes that chunk to 
somewhere else on the disk, and somehow gets it down to one atomic 
operation where it simply changes the file to point to the new location 
on disk.  Which means if you have a filesystem that is physically laid 
out on disk like this (for simplicity, assume it only has a single file):

# is data
* is also data
- is free space

######*****########--------------

When you try to write in the middle (the '*' chars) -- let's say we're 
changing them to '%' chars, this happens:

######*****########%%%%%---------

Once that's done, the file is updated so that the middle of it points to 
the fragment in the new location, and the old location is freed:

######-----########%%%%%---------



Keep in mind, because of lazy writes, it's much more likely for the 
whole change to happen at once.  Here's another example:

#####------------

Let's say we just want to overwrite the file with another one of the 
same length:

#####%%%%%-------

then, commit the transaction:

-----%%%%%-------

You see the problem?  You've now split the free space in half. 
Realistically, of course, it wouldn't be by halves, but you're basically 
inserting random air holes all over the place, and your FS is becoming 
more like foam, taking up more of the free space, until you can no 
longer use the free space....  In the above example, if we then have to 
come write some huge file, it looks like this:

*****%%%%%*******

Split right in half.  Now imagine this effect multiplied by hundreds or 
thousands of files, over time...

This is why Reiser4 needs a repacker.  While it's fine for larger files 
-- I believe after a certain point, it will write twice, so looking at 
our first example:


######*****########--------------

Write to a new, temporary place:

######*****########%%%%%---------

Write back to the original place:

######%%%%%########%%%%%---------

Complete the transaction and free the temporary space:

######%%%%%########--------------


This technique is what other journaling filesystems use, and it also 
means that writing is literally twice as slow as on a non-journaling 
filesystem, or on one with wandering logs like Reiser4.  But, it's a 
practical necessity when you're dealing with some 300 gig MySQL database 
of which only small 10k chunks are changing.  Taking twice as long on a 
10k chunk won't kill anyone, but fragmenting your 300 gig database (on a 
320 gig partition) will kill your performance, and will be very 
difficult to defragment.

But on smaller files, it would be very beneficial if we could allow the 
FS to slowly fragment (to foam-ify, if you will) and defrag once a week. 
  The amount of speed gained in each write -- and read, if it's not 
getting too awful during that week -- definitely makes up for having to 
spend an hour or so defragmenting, especially if the FS can be online at 
the time.

And you can probably figure out an optimal time to wait before 
defragmenting, since your biggest fragmentation problems happen when the 
chunk of contiguous space at the end of the disk disappears, and all of 
your free space is scattered (fragmented) throughout the disk.

Anyway, that's why.  If you disable the wandering log behavior, your 
write performance drops in half.  If you don't have a repacker, your FS 
becomes very fragmented, very fast.

I apologize for my poor ASCII art, especially if I'm dead wrong...


> No Linux file system that I'm aware of has a defragmentor, but they DO
> become fragmented, just not near as bad as FAT32 used to when MS created
> their defragmentor. The highest "non-contiguous" percent I've seen with
> EXT3 is about 12%, FAT32 I have seen over 50%, and NTFS over 30%. In

I'd like to see some numbers on Reiser4, then.  Maybe a formal 
fragmentation benchmark?

Re: reiser4 status (correction)

[Mike Benoit]

[-- Attachment #1: Type: text/plain, Size: 8997 bytes --]

Your detailed explanation is appreciated David and while I'm far from a
file system expert, I believe you've overstated the negative effects
somewhat.

It sounds to me like you've gotten Reiser4's allocation process in
regards to wandering logs correct, from what I've read anyways, but I
think you've overstated its fragmentation disadvantage when compared
against other file systems.

I think the thing we need to keep in mind here is that fragmentation
isn't always a net loss. Depending on the workload, fragmentation (or at
least not tightly packing data) could actually be a gain. In cases where
you have files (like log files or database files) that constantly grow
over a long period of time, packing them tightly at regularly scheduled
intervals (or at all?) could cause more harm then good. 

Consider this scenario of two MySQL tables having rows inserted to each
one simultaneously, and lets also assume that the two tables were
tightly packed before we started the insert process.

1 = Data for Table1
2 = Data for Table2 

Tightly packed:

111111111111222222222222----------------------------

Simultaneous inserts start:

1111111111112222222222221122112211221122------------

I believe this is actually what is happening to me with ReiserV3 on my
MythTV box. I have two recordings running at the same time, each writing
data at about 500kb/s and once the drive has less then 10% free the
whole machine grinds to a screeching halt while it attempts to find free
space. The entire 280GB drive is full of files fragmented like this.
 
Allocate on flush alone would probably help this scenario immensely. 

The other thing you need to keep in mind is that database files are like
their own little mini-file system. They have their own fragmentation
issues to deal with (especially PostgreSQL). So in cases like you
described where you are overwriting data in the middle of a file,
Reiser4 may be poor at doing this specific operation compared to other
file systems, but just because you overwrite a row that appears to be in
the middle of a table doesn't mean that the data itself is actually in
the middle of the table. If your original row is 1K, and you try to
overwrite it with 4K of data, it most likely will be put at the end of
the file anyways, and the original 1K of data will be marked for
overwriting later on. Isn't this what myisampack is for?

So while I think what you described is ultimately correct, I believe
extreme negative effects from it to be a corner case, and probably not
representative of the norm. I also believe that other Reiser4
improvements would outweigh this draw back to wandering logs, again in
average workloads. 

So the original point that I was trying to make comes back to the fact
that I don't believe Reiser4 _needs_ a repacker to maintain decent
performance. The fact that it will have a repacker just makes it that
much better for people who might need it. If Hans didn't think he could
make money off it, it probably wouldn't be so high on his priority list?
We can't fault him for that though.

Like you mentioned, if Reiser4 performance gets so poor without the
repacker, and Hans decides to charge for it, I think that will turn away
a lot potential users as they could feel that this is a type of
extortion. Get them hooked on something that only performs well for a
certain amount of time, then charge them money to keep it up. I also
think the community would write their own repacker pretty quick in
response. 

A much better approach in my opinion would be to have Reiser4 perform
well in the majority of cases without the repacker, and sell the
repacker to people who need that extra bit of performance. If I'm not
mistaken this is actually Hans intent. If Reiser4 does turn out to
perform much worse over time, I would expect Hans would consider it a
bug or design flaw and try to correct the problem however possible. 

But I guess only time will tell if this is true or not. ;)

On Fri, 2006-07-21 at 17:40 -0500, David Masover wrote:
> Maybe not XFS, but in any case, Reiser4 fragments more because of how 
> its journaling works.  It's the wandering logs.
> 
> Basically, when most Linux filesystems allocate space, they do try to 
> allocate it contiguously, and it generally stays in the same place. 
> With ext3, if you write to the middle of a file, or overwrite the entire 
> file, you're generally going to see your writes be written once to the 
> journal, and then again to the same place the file originally was.
> 
> Similarly, if you delete and then create a bunch of small files, you're 
> generally going to see the new files created in the same place the old 
> files were.
> 
> With Reiser4, wandering logs means that rather than write to the 
> journal, if you write to the middle of the file, it writes that chunk to 
> somewhere else on the disk, and somehow gets it down to one atomic 
> operation where it simply changes the file to point to the new location 
> on disk.  Which means if you have a filesystem that is physically laid 
> out on disk like this (for simplicity, assume it only has a single file):
> 
> # is data
> * is also data
> - is free space
> 
> ######*****########--------------
> 
> When you try to write in the middle (the '*' chars) -- let's say we're 
> changing them to '%' chars, this happens:
> 
> ######*****########%%%%%---------
> 
> Once that's done, the file is updated so that the middle of it points to 
> the fragment in the new location, and the old location is freed:
> 
> ######-----########%%%%%---------
> 
> Keep in mind, because of lazy writes, it's much more likely for the 
> whole change to happen at once.  Here's another example:
> 
> #####------------
> 
> Let's say we just want to overwrite the file with another one of the 
> same length:
> 
> #####%%%%%-------
> 
> then, commit the transaction:
> 
> -----%%%%%-------
> 
> You see the problem?  You've now split the free space in half. 
> Realistically, of course, it wouldn't be by halves, but you're basically 
> inserting random air holes all over the place, and your FS is becoming 
> more like foam, taking up more of the free space, until you can no 
> longer use the free space....  In the above example, if we then have to 
> come write some huge file, it looks like this:
> 
> *****%%%%%*******
> 
> Split right in half.  Now imagine this effect multiplied by hundreds or 
> thousands of files, over time...
> 
> This is why Reiser4 needs a repacker.  While it's fine for larger files 
> -- I believe after a certain point, it will write twice, so looking at 
> our first example:
> 
> 
> ######*****########--------------
> 
> Write to a new, temporary place:
> 
> ######*****########%%%%%---------
> 
> Write back to the original place:
> 
> ######%%%%%########%%%%%---------
> 
> Complete the transaction and free the temporary space:
> 
> ######%%%%%########--------------
> 
> 
> This technique is what other journaling filesystems use, and it also 
> means that writing is literally twice as slow as on a non-journaling 
> filesystem, or on one with wandering logs like Reiser4.  But, it's a 
> practical necessity when you're dealing with some 300 gig MySQL database 
> of which only small 10k chunks are changing.  Taking twice as long on a 
> 10k chunk won't kill anyone, but fragmenting your 300 gig database (on a 
> 320 gig partition) will kill your performance, and will be very 
> difficult to defragment.
> 
> But on smaller files, it would be very beneficial if we could allow the 
> FS to slowly fragment (to foam-ify, if you will) and defrag once a week. 
>   The amount of speed gained in each write -- and read, if it's not 
> getting too awful during that week -- definitely makes up for having to 
> spend an hour or so defragmenting, especially if the FS can be online at 
> the time.
> 
> And you can probably figure out an optimal time to wait before 
> defragmenting, since your biggest fragmentation problems happen when the 
> chunk of contiguous space at the end of the disk disappears, and all of 
> your free space is scattered (fragmented) throughout the disk.
> 
> Anyway, that's why.  If you disable the wandering log behavior, your 
> write performance drops in half.  If you don't have a repacker, your FS 
> becomes very fragmented, very fast.
> 
> I apologize for my poor ASCII art, especially if I'm dead wrong...
> 
> 
> > No Linux file system that I'm aware of has a defragmentor, but they DO
> > become fragmented, just not near as bad as FAT32 used to when MS created
> > their defragmentor. The highest "non-contiguous" percent I've seen with
> > EXT3 is about 12%, FAT32 I have seen over 50%, and NTFS over 30%. In
> 
> I'd like to see some numbers on Reiser4, then.  Maybe a formal 
> fragmentation benchmark?
-- 
Mike Benoit <ipso@snappymail.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[David Masover]

Mike Benoit wrote:
> Your detailed explanation is appreciated David and while I'm far from a
> file system expert, I believe you've overstated the negative effects
> somewhat.
> 
> It sounds to me like you've gotten Reiser4's allocation process in
> regards to wandering logs correct, from what I've read anyways, but I
> think you've overstated its fragmentation disadvantage when compared
> against other file systems.
> 
> I think the thing we need to keep in mind here is that fragmentation
> isn't always a net loss. Depending on the workload, fragmentation (or at
> least not tightly packing data) could actually be a gain. In cases where

defragmented != tightly packed.

> you have files (like log files or database files) that constantly grow
> over a long period of time, packing them tightly at regularly scheduled
> intervals (or at all?) could cause more harm then good. 

This is true...

> Consider this scenario of two MySQL tables having rows inserted to each
> one simultaneously, and lets also assume that the two tables were
> tightly packed before we started the insert process.
> 
> 1 = Data for Table1
> 2 = Data for Table2 
> 
> Tightly packed:
> 
> 111111111111222222222222----------------------------
> 
> Simultaneous inserts start:
> 
> 1111111111112222222222221122112211221122------------
> 

> Allocate on flush alone would probably help this scenario immensely. 

Yes, it would.  You'd end up with

1111111111112222222222221111111122222222------------

assuming they both fit into RAM.  And of course they could later be 
repacked.

By the way, this is the NTFS approach to avoiding fragmentation -- try 
to avoid fragmenting anything below a certain block size.  I, for one, 
would be perfectly happy if my large files were split up every 50 or 100 
megs or so.

The problem is when you get tons of tiny files and metadata stored so 
horribly inefficiently that things like Native Command Queuing is 
actually a huge performance boost.

> The other thing you need to keep in mind is that database files are like
> their own little mini-file system. They have their own fragmentation
> issues to deal with (especially PostgreSQL).

I'd rather not add to that.  This is one reason to hate virtualization, 
by the way -- it's bad enough to have a fragmented NTFS on your Windows 
installation, but worse if the disk itself is a fragmented sparse file 
on Linux.

> So in cases like you
> described where you are overwriting data in the middle of a file,
> Reiser4 may be poor at doing this specific operation compared to other
> file systems, but just because you overwrite a row that appears to be in
> the middle of a table doesn't mean that the data itself is actually in
> the middle of the table. If your original row is 1K, and you try to
> overwrite it with 4K of data, it most likely will be put at the end of
> the file anyways, and the original 1K of data will be marked for
> overwriting later on. Isn't this what myisampack is for?

If what you say is true, isn't myisampack also an issue here?  Surely it 
doesn't write out an entirely separate copy of the file?

Anyway, the most common usage I can see for mysql would be overwriting a 
1K row with another 1K row, or dropping a row, or adding a wholly new 
row.  I may be a bit naive here...

But then, isn't there also some metadata somewhere which says things 
like how many rows you have in a given table?

And it's not just databases.  Consider BitTorrent.  The usual BitTorrent 
way of doing things is to create a sparse file, then fill it in randomly 
as you receive data.  Only if you decide to allocate the whole file 
right away, instead of making it sparse, you gain nothing on Reiser4, 
since writes will be just as fragmented as if it was sparse.

Personally, I'd rather leave it as sparse, but repack everything later.

> So while I think what you described is ultimately correct, I believe
> extreme negative effects from it to be a corner case, and probably not
> representative of the norm. I also believe that other Reiser4
> improvements would outweigh this draw back to wandering logs, again in
> average workloads. 

Depends on your definition of average.  I'm also speaking from 
experience.  On Gentoo, /usr/portage started out being insanely fast on 
Reiser4, because it barely had to seek at all -- despite being about 
145,000 small files.  I think it was maybe half that when I first put it 
on r4, but it's more than twice as slow now, and you can hear it thrashing.

Now, the wandering logs did make the rsync process pretty fast -- the 
entire thing gets rsync'd against one of the Gentoo mirrors.  For anyone 
using Debian, this is the equivalent of "apt-get update".

Only now, this rsync process is not only entirely disk-bound, it's 
something like 10x as slow.  I have a gig of RAM, so at least it's fast 
once it's cached, but it's obviously horrendously fragmented.  I am not 
sure if it's individual files or directories, but it could REALLY use a 
repack.

 From what I remember of v3, it was never quite this bad, but it never 
started out as fast as it did on Reiser4.

This is why I'm curious to see some benchmarks, by the way -- all of 
this is subjective, and from memory.

> Like you mentioned, if Reiser4 performance gets so poor without the
> repacker, and Hans decides to charge for it, I think that will turn away
> a lot potential users as they could feel that this is a type of
> extortion. Get them hooked on something that only performs well for a
> certain amount of time, then charge them money to keep it up. I also
> think the community would write their own repacker pretty quick in
> response. 

Depends.  Unfortunately, it's far more likely that the community would 
go "fsck this" and use XFS instead.  Or JFS.  Or any of the other 
filesystems that Linux has which don't need a repacker.

It would eventually get done by the community, but if it's taking the 
Namesys guys this long, and if they really expect to be able to make 
money off of it, it must not be as trivial as I think it is.

> A much better approach in my opinion would be to have Reiser4 perform
> well in the majority of cases without the repacker, and sell the
> repacker to people who need that extra bit of performance. If I'm not
> mistaken this is actually Hans intent.

Hans?

> If Reiser4 does turn out to
> perform much worse over time, I would expect Hans would consider it a
> bug or design flaw and try to correct the problem however possible. 

Or a design constraint...

> But I guess only time will tell if this is true or not. ;)

I'll tell you now it's true.

To be fair, I'm not entirely up to date, but I've had a Reiser4 root 
partition for over a year now.  It seems pretty decent for most things, 
but I've definitely noticed that anywhere like /usr/portage -- lots of 
files changing, lots staying the same, over time -- ends up pretty badly 
fragmented.  Other examples would be games, especially Steam games and 
MMOs, played using Wine.

And I'd like some benchmarks, but I strongly suspect that this problem 
is pretty bad -- and that the more you'd think a particular workload is 
suited for Reiser4, the better the benchmarks are initially, the worse 
it will degrade if there's any writing going on.

Re: reiser4 status (correction)

[Hans Reiser]

David Masover wrote:

>
>
> And it's not just databases.  Consider BitTorrent.  The usual
> BitTorrent way of doing things is to create a sparse file, then fill
> it in randomly as you receive data.  Only if you decide to allocate
> the whole file right away, instead of making it sparse, you gain
> nothing on Reiser4, since writes will be just as fragmented as if it
> was sparse.

If you don't flush it before you fill it, then in reiser4 it does not
matter if it starts out sparse.

>
> Personally, I'd rather leave it as sparse, but repack everything later.

We have 3 levels of optimization: 1) at each modification, 2) at each
flush, and 3) at each repack.  Each of these operates on a different
time scale, and all 3 are worthy of doing as right as we can.

Now, the issue of where should airholes be?  Why, that is the grand
experiment that will start to happen in a few months.  Nobody knows yet
what defaults we should have, and whatever we choose, there will be some
users who gain from explicit control of it.

>   it must not be as trivial as I think it is.

The problem is that there was a list of must dos, and this was just one
of them.  If reiser4 goes in, then fsync is the only thing in front of
the repacker.  The list has reduced in size a bunch.

>
>> A much better approach in my opinion would be to have Reiser4 perform
>> well in the majority of cases without the repacker, and sell the
>> repacker to people who need that extra bit of performance. If I'm not
>> mistaken this is actually Hans intent.
>
>
> Hans?

Yes, that's the idea.  Only sysadmins of large corps are likely to buy. 
We throw in service and support as well for those who purchase it.

If I was making money, I would not do this, but I am not.  I am not
really willing to work a day job for the rest of my life supporting guys
in Russia, it is only ok to do as a temporary measure.  I am getting
tired....

>
>> If Reiser4 does turn out to
>> perform much worse over time, I would expect Hans would consider it a
>> bug or design flaw and try to correct the problem however possible. 
>
>
I would want Reiser4 without a repacker to outperform all other
filesystems.  The problem with this fragmentation over time issue is
that it is hard to tweak allocation, measure the effect, tweak again,
etc.  Not sure how to address it without a lot of work.  Maybe we need
to create some sort of condensed portage benchmark....

Hans

Hans

Re: reiser4 status (correction)

[Mike Benoit]

[-- Attachment #1: Type: text/plain, Size: 4501 bytes --]

On Fri, 2006-07-21 at 23:53 -0600, Hans Reiser wrote:
> We have 3 levels of optimization: 1) at each modification, 2) at each
> flush, and 3) at each repack.  Each of these operates on a different
> time scale, and all 3 are worthy of doing as right as we can.
> 
> Now, the issue of where should airholes be?  Why, that is the grand
> experiment that will start to happen in a few months.  Nobody knows yet
> what defaults we should have, and whatever we choose, there will be some
> users who gain from explicit control of it.
> 

Wouldn't the most recently modified files give a good hint to the
repacker? The files larger then a few kilobytes most recently modified
would be packed with air holes, and the files least recently modified
would be packed more tightly? Perhaps even files that are currently the
most fragmented would be repacked with air holes, as compared to files
least fragmented would be packed more tightly.

Could you not also write a small little app that gathers all kinds of
stats about a file system and sends it to a Namesys server in hopes of
finding better statistical data? I'm sure there are thousands of users
that would be willing to run this app for the "greater good" regardless
or not if they used ReiserFS in the first place. Things like the number
of files on the disk, what percentage of those files have been modified
in the last week, which files are the most/least fragmented, and when
they were last modified, etc...?

> >
> >> A much better approach in my opinion would be to have Reiser4 perform
> >> well in the majority of cases without the repacker, and sell the
> >> repacker to peopleisn' something  who need that extra bit of performance. If I'm not
> >> mistaken this is actually Hans intent.
> >
> >
> > Hans?
> 
> Yes, that's the idea.  Only sysadmins of large corps are likely to buy. 
> We throw in service and support as well for those who purchase it.
> 
> If I was making money, I would not do this, but I am not.  I am not
> really willing to work a day job for the rest of my life supporting guys
> in Russia, it is only ok to do as a temporary measure.  I am getting
> tired....
> 

Personally, as much as I would like it all to be free, I think I would
be much more willing to pay for compression/encryption (on both servers
and desktops) then I would be for a repacker. Hard disks cost money, and
if I can compress the vast majority of my data and save on purchasing a
new hard disk, that is well worth it. I also have some important data
that I would really like to encrypt, which is also worth spending money
on. But gaining ~10% in performance probably isn't worthwhile spending
money on as I most likely wouldn't notice a difference in my day to day
life, unless my server was incredibly busy. 

There is no doubt there is a market for a repacker, but I think people
are much more likely to spend money on something that is immediately
tangible, like disk space instantly being free'd up by compression, or
data instantly being encrypted. As compared to something that is much
less tangible like fragmentation percentages and minor I/O throughput
improvements. I used to work at a large, world wide web hosting company
and I could see making a case to management for purchasing Reiser4
compression would be pretty easy for our shared servers. Instantly
freeing up large amounts of disk space (where .html/.php files were the
vast majority) would save huge amounts of money on disk drives,
especially since most of the servers used RAID1 and adding new drives
was a huge pain in the neck. Making a case to purchase a repacker would
be much, much more difficult.

See, customers who used lots of CPU were easy to up-sell to a dedicated
server because page load times were tangible and if they didn't move we
would be forced to shut them off. However customers who used gobs of
disk space were much more difficult to up-sell to dedicated servers
because it didn't affect themselves or other customers in a tangible
way. They wouldn't notice any difference by moving to a much more
expensive dedicated server.

I would like to see Namesys succeed and become incredibly profitable for
Hans, if nothing else for the fact that he has given a huge amount to
the open source community already. A profitable Namesys only means we'll
have a greater chance of seeing even more interesting stuff from them in
the future. 

-- 
Mike Benoit <ipso@snappymail.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[David Masover]

Mike Benoit wrote:


> Could you not also write a small little app that gathers all kinds of
> stats about a file system and sends it to a Namesys server in hopes of
> finding better statistical data? I'm sure there are thousands of users

Assuming the results are all made available, essentially public domain. 
  If it becomes "improve Reiser" instead of "improve filesystems", then 
only fans of Reiser will do it.

>>>> A much better approach in my opinion would be to have Reiser4 perform
>>>> well in the majority of cases without the repacker, and sell the
>>>> repacker to peopleisn' something  who need that extra bit of performance. If I'm not
>>>> mistaken this is actually Hans intent.
>>>
>>> Hans?
>> Yes, that's the idea.  Only sysadmins of large corps are likely to buy. 
>> We throw in service and support as well for those who purchase it.

> Personally, as much as I would like it all to be free, I think I would
> be much more willing to pay for compression/encryption (on both servers
> and desktops) then I would be for a repacker. Hard disks cost money, and
> if I can compress the vast majority of my data and save on purchasing a
> new hard disk, that is well worth it. I also have some important data
> that I would really like to encrypt, which is also worth spending money
> on. But gaining ~10% in performance probably isn't worthwhile spending
> money on as I most likely wouldn't notice a difference in my day to day
> life, unless my server was incredibly busy. 

Assuming the performance gain is only 10%, you may be right.  Still, 
faster disks, controllers, buses, and CPUs also cost money.

I would be willing to pay for both, even, if the price was reasonable (I 
think there was a scheme based on amount of space placed under the FS?), 
and if I could be guaranteed updates (bug fixes, new kernels, etc) for 
at least the functionality that I've paid for, with no additional cost.

Reiser4 lazy writes make a huge difference on a laptop, and my current 
laptop is a Mac.  That means that around when the next Mac OS rolls out, 
it will be perfectly reasonable to spend some $50 or $100 to make my 
Linux faster instead of my Mac OS, especially since I'm trying to 
migrate off of Mac OS as much as possible.  (I miss package management.)

One more suggestion:  Maybe make it free for non-commercial use only? 
And by that I mean, based on how the FS is actually being used.  I don't 
know if piracy is or has been a problem for Namesys, but at least the 
economics of it makes sense:  A fifteen year old hacker won't want to 
pay for an FS, but might have a lot to contribute.  But if your main 
market is large servers, have those people pay -- they are running their 
business off your FS.

> There is no doubt there is a market for a repacker, but I think people
> are much more likely to spend money on something that is immediately
> tangible, like disk space instantly being free'd up by compression, or
> data instantly being encrypted. As compared to something that is much

The compression will probably mostly be about speed.  Remember, if we're 
talking about people who want to see tangible, visceral results, we're 
probably also talking about end-users.  And trust me, the vast majority 
of most of my data (as an end-user) is not very compressible.

Ok, I lied:  I love games, and you can make a very modern-looking game 
in 96K:

http://produkkt.abraxas-medien.de/kkrieger

But while that could be seen as a kind of compression, it's done by 
hand, and is fairly irrelevant to filesystems.

No, mostly we're talking about things like office documents, the 
majority of which fit in less than a gigabyte, and multimedia (music, 
movies, games) which will gain very little from compression.  If 
anything, the benefit would be mostly in compressing software.

> less tangible like fragmentation percentages and minor I/O throughput
> improvements. I used to work at a large, world wide web hosting company
> and I could see making a case to management for purchasing Reiser4
> compression would be pretty easy for our shared servers. Instantly
> freeing up large amounts of disk space (where .html/.php files were the
> vast majority) would save huge amounts of money on disk drives,
> especially since most of the servers used RAID1 and adding new drives
> was a huge pain in the neck. Making a case to purchase a repacker would
> be much, much more difficult.

Hmm, the problem is, if storage space is really the big deal, it's been 
done before, and some of these efforts are still usable and free:

http://parallel.vub.ac.be/~johan/compFUSEd/
http://www.miio.net/fusecompress/
http://north.one.pl/~kazik/pub/LZOlayer/
http://apfs.humorgraficojr.com/apfs_ingles.html

And while we're on the topic, here's an FS that does unpacking of 
archives, probably about the same way we imagined it in Reiser4 
pseudofiles/magic:

http://www.nongnu.org/unpackfs/

But regardless, as far as I can tell, the only real, tangible benefit of 
using Reiser4 compression instead of one of those four FUSE filesystems 
is speed.  Reiser4 would compress/decompress when actually hitting the 
disk, not just the FS, and it would also probably use in-kernel 
compression, rather than calling out to userspace on every FS operation.

But you see, if you're talking about speed, 10% is a respectably big 
improvement, so I could see selling them on a repacker at the same time.

Maybe bundles are a good idea...
Maybe there should be a Reiser4 Whitepaper Value Pack, once everything 
on the whitepaper is done?

> See, customers who used lots of CPU were easy to up-sell to a dedicated
> server because page load times were tangible and if they didn't move we
> would be forced to shut them off. However customers who used gobs of
> disk space were much more difficult to up-sell to dedicated servers
> because it didn't affect themselves or other customers in a tangible
> way. They wouldn't notice any difference by moving to a much more
> expensive dedicated server.

Sounds like more a marketing problem than a technical one.  Couldn't you 
just charge more on the virtual server?  Or start charging by the megabyte?

> I would like to see Namesys succeed and become incredibly profitable for
> Hans, if nothing else for the fact that he has given a huge amount to
> the open source community already. A profitable Namesys only means we'll
> have a greater chance of seeing even more interesting stuff from them in
> the future. 

Amen.

Re: reiser4 status (correction)

[Mike Benoit]

[-- Attachment #1: Type: text/plain, Size: 7001 bytes --]

On Sat, 2006-07-22 at 07:34 -0500, David Masover wrote:

> The compression will probably mostly be about speed.  Remember, if
> we're 
> talking about people who want to see tangible, visceral results,
> we're 
> probably also talking about end-users.  And trust me, the vast
> majority 
> of most of my data (as an end-user) is not very compressible.
> 

Sure, mine too. Between the many gigs of MP3s and movies I have stored
on my HD, only about 10-20GB is the OS, Email, and documents/source
code. Even just compressing that small portion though I could probably
save between 5-10GB. The difference is though I can do a df before, and
a df after, and I can instantly see I got my moneys worth. Same with
encryption. 

With the repacker it is much more difficult (for average users) to time
how long a program takes to load some file (or launch), before the
repacker and after. Especially since caching comes in to play. 

Also, according to this little poll on one of the compressed FUSE sites
you linked to, more people are looking to compression for space saving,
then for speed:
http://parallel.vub.ac.be/~johan/compFUSEd/index.php?option=polls&task=results&pollid=31


> 
> No, mostly we're talking about things like office documents, the 
> majority of which fit in less than a gigabyte, and multimedia (music, 
> movies, games) which will gain very little from compression.  If 
> anything, the benefit would be mostly in compressing software.
> 
> > less tangible like fragmentation percentages and minor I/O throughput
> > improvements. I used to work at a large, world wide web hosting company
> > and I could see making a case to management for purchasing Reiser4
> > compression would be pretty easy for our shared servers. Instantly
> > freeing up large amounts of disk space (where .html/.php files were the
> > vast majority) would save huge amounts of money on disk drives,
> > especially since most of the servers used RAID1 and adding new drives
> > was a huge pain in the neck. Making a case to purchase a repacker would
> > be much, much more difficult.
> 
> Hmm, the problem is, if storage space is really the big deal, it's been 
> done before, and some of these efforts are still usable and free:
> 
> http://parallel.vub.ac.be/~johan/compFUSEd/
> http://www.miio.net/fusecompress/
> http://north.one.pl/~kazik/pub/LZOlayer/
> http://apfs.humorgraficojr.com/apfs_ingles.html
> 
> And while we're on the topic, here's an FS that does unpacking of 
> archives, probably about the same way we imagined it in Reiser4 
> pseudofiles/magic:
> 
> http://www.nongnu.org/unpackfs/
> 
> But regardless, as far as I can tell, the only real, tangible benefit of 
> using Reiser4 compression instead of one of those four FUSE filesystems 
> is speed.  Reiser4 would compress/decompress when actually hitting the 
> disk, not just the FS, and it would also probably use in-kernel 
> compression, rather than calling out to userspace on every FS operation.
> 
> But you see, if you're talking about speed, 10% is a respectably big 
> improvement, so I could see selling them on a repacker at the same time.
> 

Most of those FUSE file systems you linked to scare me. This is from the
APFS web page:

"Permissions do not work!
Do not worry, it is not your fault, permissions are not implemented yet.
It is like all files had rwxrwxrwx mode.

I have lost all my data! How do I get it back?
From the backup, obviously."

FUSE is great, but can it even come close to matching the performance of
in-kernel file systems? Not only that, but if you want to compress a
directory you have to go through about a 12 step process of moving the
files, setting up a mount point, and moving the files back. 

Will Reiser4 not allow us to mount with compression enabled then
enable/disable compression on a per file/per directory basis? 


> Maybe bundles are a good idea...
> Maybe there should be a Reiser4 Whitepaper Value Pack, once everything 
> on the whitepaper is done?
> 
> > See, customers who used lots of CPU were easy to up-sell to a dedicated
> > server because page load times were tangible and if they didn't move we
> > would be forced to shut them off. However customers who used gobs of
> > disk space were much more difficult to up-sell to dedicated servers
> > because it didn't affect themselves or other customers in a tangible
> > way. They wouldn't notice any difference by moving to a much more
> > expensive dedicated server.
> 
> Sounds like more a marketing problem than a technical one.  Couldn't you 
> just charge more on the virtual server?  Or start charging by the megabyte?

You could, and we did charge by the megabyte, only after they exceeded
the limit of their package. However web hosting is a fiercely
competitive market, so we were constantly adjusting our packages to
include more disk space, more bandwidth, more features for the same
amount, or lower prices, just to compete. The way many "shared hosting"
companies work is each server is waaay over sold, at least in the disk
space department. We would have anywhere from 500-1500 accounts on one
server, usually with only one or two 80-120GB SCSI RAID1 arrays, if the
average package offered 500MB of disk space, the vast majority of
customers couldn't even come close to reaching their maximum disk usage.
Thats pretty much the only way you make money with $10-20/month
packages.

If you add up the cost of SCSI drives (x2 for RAID), SCSI controllers,
replacement costs every 2-3yrs, the time to install them, you could
easily charge $100/server for the compression plugin. It would pretty
much be a no brainer for a huge number of the web hosting companies out
there to go this route. The company I worked for probably wouldn't blink
an eye at dropping $5K to equip just one of its many data centers. If
Namesys offered "site licensing", I could see installing compression on
our customers dedicated servers as a value added service that they could
profit from also.

Compression would be a huge win for these companies, especially since
90% of the data on these servers is HTML/PHP/PERL files that change very
rarely. 

I don't doubt the benefits of the repacker, but from a business
perspective the repacker is something that runs transparently in the
background, once you install it, things magically speed up then you
never hear from it again as it does its job. Out of sight, out of mind.
Whereas the compression/encryption plugin are always in your face, every
time you run df, or enter a passphrase to gain access to your files, you
know its there and working. Its something you'll tell your friends
about. After you first run it, the repacker just fades away in to the
background and you forget about it.

Charging only for commercial use of the repacker/compression/encryption
plugin would be a great middle ground. 

-- 
Mike Benoit <ipso@snappymail.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 status (correction)

[David Masover]

Mike Benoit wrote:

> code. Even just compressing that small portion though I could probably
> save between 5-10GB. The difference is though I can do a df before, and
> a df after, and I can instantly see I got my moneys worth. Same with
> encryption. 

In the case of encryption, it's also got competition.  There are two 
FUSE filesystems that do crypto, and there's cryptoloop/dm-crypt, which 
you need anyway for encrypted swap.

True, it's nowhere near as nice, but it is functional.  You would think 
one of these would be easier to develop to where it's useful.  And if 
you're encrypting it, you're already accepting a performance hit.

So again, the Reiser4 advantage here is all speed.

> http://parallel.vub.ac.be/~johan/compFUSEd/index.php?option=polls&task=results&pollid=31

Of course they are, or they wouldn't use FUSE.

But if that's really true, why wouldn't a FUSE driver help you?

> Most of those FUSE file systems you linked to scare me. This is from the
> APFS web page:

> I have lost all my data! How do I get it back?
> From the backup, obviously."

To be fair, that's what you get from any FS.

But how hard would it be to make a FUSE filesystem work properly?  How 
hard would it be to get a Reiser4 plugin to work properly?

> FUSE is great, but can it even come close to matching the performance of
> in-kernel file systems?

Performance again!

> Not only that, but if you want to compress a
> directory you have to go through about a 12 step process of moving the
> files, setting up a mount point, and moving the files back. 

You listed 3 steps.  Besides, I don't see anything stopping you from 
modifying one of these to selectively compress things, the way Reiser4 
would.  So, put your entire FS under a FUSE system, then configure which 
directories you want compressed.

Again, the drawback is huge gobs of performance.  It just seems natural 
to use the repacker if you're going to use any Reiser4-based replacement 
for these.

> You could, and we did charge by the megabyte, only after they exceeded
> the limit of their package. However web hosting is a fiercely
> competitive market, so we were constantly adjusting our packages to
> include more disk space, more bandwidth, more features for the same
> amount, or lower prices, just to compete. The way many "shared hosting"
> companies work is each server is waaay over sold, at least in the disk

Ah, so I guess the advantage of a more expensive, dedicated box is that 
you know no one's overselling it?

Personally, I get a little sick of the insane amounts of overselling 
that happen -- you just know it's going to come back and bite you in the 
ass.  This is happening right now with bandwidth, and overselling is 
pretty much solely responsible for the whole Net Neutrality mess -- it 
would be a complete non-issue if "5 megabits to your house!!!!!" was 
backed up by an actual 5 megabits reserved for me, or if they sold them 
as "2-5 megabits", where 2 is guaranteed, and 5 is what you get when no 
one else is using it.

I'm a bit skeptical of my local ISP's Fiber To The Home initiative, 
because I can't imagine they have enough spare upstream bandwidth just 
lying around.

I mean, if you actually have enough bandwidth, you don't want or need 
your ISP to do QoS or prioritizing for you -- you can just do it 
yourself.  Personally, ssh packets would be of a much higher priority to 
me than anything else...

> Thats pretty much the only way you make money with $10-20/month
> packages.

Yeah, I know, hard for an honest guy to compete...

> I don't doubt the benefits of the repacker, but from a business
> perspective the repacker is something that runs transparently in the
> background, once you install it, things magically speed up then you
> never hear from it again as it does its job. Out of sight, out of mind.
> Whereas the compression/encryption plugin are always in your face, every
> time you run df, or enter a passphrase to gain access to your files, you
> know its there and working. Its something you'll tell your friends
> about. After you first run it, the repacker just fades away in to the
> background and you forget about it.

Ah.  This might explain the success of things like Ruby On Rails.  Once 
it's done, the insane amount of CPU required to run a high-level 
interpreted language (versus even something semi-interpreted like perl) 
is out of sight, out of mind.  But when you first set it up, the savings 
in development time are immediately obvious, in your face.

Although I would want to find a way to avoid typing a passphrase every 
time -- maybe keep the key on a USB keychain.  Passphrases would just 
annoy users.  Unless, of course, you could tie it to logon -- assuming 
you can actually change the passphrase later...

> Charging only for commercial use of the repacker/compression/encryption
> plugin would be a great middle ground. 

Good, I'm glad it wasn't a completely insane idea.

Re: reiser4 status (correction)

[Hans Reiser]

Mike Benoit wrote:

>On Sat, 2006-07-22 at 07:34 -0500, David Masover wrote:
>
>  
>
>>The compression will probably mostly be about speed.  Remember, if
>>we're 
>>talking about people who want to see tangible, visceral results,
>>we're 
>>probably also talking about end-users.  And trust me, the vast
>>majority 
>>of most of my data (as an end-user) is not very compressible.
>>
>>    
>>
>
>Sure, mine too. Between the many gigs of MP3s and movies I have stored
>on my HD, only about 10-20GB is the OS, Email, and documents/source
>code. Even just compressing that small portion though I could probably
>save between 5-10GB. The difference is though I can do a df before, and
>a df after, and I can instantly see I got my moneys worth. Same with
>encryption. 
>  
>
I am looking forward to the first user email complaining that he
compressed a file stored on reiser4 and it didn't save space (and then
someday maybe even an email saying that he compressed a file and it took
up more space but the user space compressor is sure that it saved space
and he does not understand....).:)

>With the repacker it is much more difficult (for average users) to time
>how long a program takes to load some file (or launch), before the
>repacker and after.
>
I think you are confusing repacking and compression?  Repacking, by
removing seeks, will make it more predictable not less.

> Especially since caching comes in to play. 
>
>Also, according to this little poll on one of the compressed FUSE sites
>you linked to, more people are looking to compression for space saving,
>then for speed:
>http://parallel.vub.ac.be/~johan/compFUSEd/index.php?option=polls&task=results&pollid=31
>
>
>  
>
>>No, mostly we're talking about things like office documents, the 
>>majority of which fit in less than a gigabyte, and multimedia (music, 
>>movies, games) which will gain very little from compression.  If 
>>anything, the benefit would be mostly in compressing software.
>>
>>    
>>
>>>less tangible like fragmentation percentages and minor I/O throughput
>>>improvements. I used to work at a large, world wide web hosting company
>>>and I could see making a case to management for purchasing Reiser4
>>>compression would be pretty easy for our shared servers. Instantly
>>>freeing up large amounts of disk space (where .html/.php files were the
>>>vast majority) would save huge amounts of money on disk drives,
>>>especially since most of the servers used RAID1 and adding new drives
>>>was a huge pain in the neck. Making a case to purchase a repacker would
>>>be much, much more difficult.
>>>      
>>>
>>Hmm, the problem is, if storage space is really the big deal, it's been 
>>done before, and some of these efforts are still usable and free:
>>
>>http://parallel.vub.ac.be/~johan/compFUSEd/
>>http://www.miio.net/fusecompress/
>>http://north.one.pl/~kazik/pub/LZOlayer/
>>http://apfs.humorgraficojr.com/apfs_ingles.html
>>
>>And while we're on the topic, here's an FS that does unpacking of 
>>archives, probably about the same way we imagined it in Reiser4 
>>pseudofiles/magic:
>>
>>http://www.nongnu.org/unpackfs/
>>
>>But regardless, as far as I can tell, the only real, tangible benefit of 
>>using Reiser4 compression instead of one of those four FUSE filesystems 
>>is speed.  Reiser4 would compress/decompress when actually hitting the 
>>disk, not just the FS, and it would also probably use in-kernel 
>>compression, rather than calling out to userspace on every FS operation.
>>    
>>
I think that compressing only on flush is a big issue.  It was a lot
harder to code it, but it really makes a difference.  You don't want a
machine with a working set that fits into RAM to be compressing, that
would be lethal to performance, and it is a very important case.

Hans

portage tree (Was: Re: reiser4 status (correction))

[Christian Trefzer]

[-- Attachment #1.1: Type: text/plain, Size: 2150 bytes --]

Hi,

The portage tree is such a fine testing object since it should be sort
of a "best case scenario" for reiser filesystems, and needs no real
backup in case of a screwup during tests.

I've been on Gentoo for years now, used reiser3 since the days when you
had to patch it into a 2.2 kernel and tried out reiser4 on my portage
tree first of all. During my comparison, reiser3 + notail resulted in
huge amounts of wasted disk space, obviously it is not very smart to use
up a whole 4k block for a file of only few bytes. reiser3 with packed
tails was a lot better, but still it was a remarkable difference from
there to reiser4, and I am only talking storage efficiency here which
directly translates to the amount of I/O necessary to get the same data
all the way to RAM and CPU. The reiser4 dev team did a great job there!

With reiser3, I made another observation wrt. fragmentation, most of
which should be alleviated by reiser4 and its delayed allocation. Over
time, doing package database updates and querying for pending updates to
the local machine became slower. Packing up everything in a tarball,
mkfs, mount with -onoatime, unpack usually fixed the "problem". Since
the performance reduction in reiser4 is a lot less (I can hardly tell if
it makes a difference before vs. after the backup/restore "repacking") I
hardly do it any more, but once in a while I still use my script
initially written for the comparison test.

Attached is the shell script I used for comparing various mount options
for ReiserFS v3 and later v4. It is capable of converting between
ext2|3, reiser3|4 and xfs, provided the target fs stores the data
efficiently enough that everything fits inside. (I had "interesting"
results when moving from reiser3 to ext3 for testing, esp. wrt. portage
tree ; ) 

The script falls short at handling arbitrary mount options, but may be
trivially edited to use any desired options for a single use. Reading
and understanding is recommended before feeding your viable data to it.
I hope somebody may find it useful, or at least inspiring. Any comments
appreciated!

Kind regards,
Chris

[-- Attachment #1.2: fsconvert --]
[-- Type: text/plain, Size: 6923 bytes --]

#!/bin/sh

# ! WARNING !
# As always, BE CAREFUL AND KNOW WHAT YOU DO WHEN RUNNING THIS PROGRAM!
# At best, read the code, understand what it does, and take appropriate
# precautions. Backups of valuable data are a requirement anyway, so go 
# for it BEFORE you try this out!
#
#
# SYNOPSIS:
# fsconvert.sh <fstype> <compression method> <mountpoint>
# 
# where fstype may be any of ext[2|3], reiser[fs|4] or xfs
# and compression method is a choice of compress, gzip or bzip2.
#
#
# USE:
# This script was initially intended to re-create old reiserfs 3 
# filesystems for performance reasons, and can reasonably be considered
# useful when run once every year or two for any filesystem under heavy
# read/write load. Simply give the existing filesystem type as the fstype
# argument.
#
# Though it was written for that single purpose at first, it was recently
# adapted for filesystem type migration kind of operations, as it did
# never care about the fs type in the beginning and now takes the
# destination fs type as an argument.
#
# The script REQUIRES the concerned fs to be mounted, to have an entry in
# /etc/fstab, to have no further active mountpoints in a subdirectory,
# enough free disk space in $archdir to back up data and of course the
# required tools (mkfs.<fstype>, tar, compression tool of choice) to do
# all the magic.
#
# Please send any comment, modification, failure report etc. to
# ctrefzer AT gmx DOT de
#
# Have fun!
# Chris

# Changelog:
#
# 2005-10-03
# Enhanced commandline interface to support multiple target file system types
# and compression methods.
# Filesystems supported so far:
#  - ext2 / 3
#  - reiser 3 / 4
#  - xfs
# 
# Compression methods which seemed appropriate:
#  - none (uncompressed archive for almost no CPU load)
#  - compress (quite fast even on old machines)
#  - gzip (realtime on new machines, yet impressive I/O reduction)
#  - bzip2 (in case of tight disk space, but takes ages wrt. the others)

### Configure here:

# where to store the archives:
archdir="${PWD}/.fsconvert"


# Sane defaults for compression levels (and guidelines): use fast gzip on 
# modern CPUs where GZIP -3 gives better compression than LZW, yet still at
# realtime, whereas BZIP2 should only be used when disk space is _really_
# critical - it will take forever. LZW (compress) is advised on not-so-recent
# machines to save some I/O without slowing things down. A nifty compromise
# may be GZIP -1 on P2/P3 class machines.

export GZIP="-1"
export BZIP2="-9"


### No modifications required below here!


### Some helper functions:

function e_report {
	echo -en "${1}..."
}

function e_done {
	# Paranoia is good! It's all about the data...
	sync
	echo " Done."
}

function e_fail {
	echo " FAILED!"
	exit ${1}
}

function e_usage {
	echo "USAGE: ${0} <fstype> <compressiontype> <mountpoint>"
	exit 1
}

if [ -z ${1} ]
then
	e_usage
else
	case ${1} in
		ext2 | ext3 | reiserfs | reiser4 | xfs )
			fstype="${1}"
		;;
		
		* )
			e_usage
		;;
	esac
	
	case ${2} in
		none | compress | gzip | bzip2 )
			compression="${2}"
		;;
		
		* )
			e_usage
		;;
	esac
	
	mountpoint="`echo ${3} | sed -e\"s%/$%%\"`"
	device="`cat /etc/fstab | grep -v ^# | grep "[^[:print:]]${mountpoint}[^[:print:]]" | awk '{print \$1}'`"
	
	if [ -z "${device}" ]
	then
		echo "No device mounted to ${mountpoint} could be found in /etc/fstab!"
		exit 1
	fi
	
	if [ "$(echo ${device} | wc -l )" != "1" ]
	then
		echo "More than one device was found:"
		echo "${device}"
		exit 1
	fi
	
	case ${compression} in
		none )
			suffix="tar"
			tar_compression_flag=""
		;;
		
		compress )
			suffix="tar.Z"
			tar_compression_flag="--${compression}"
		;;
		
		gzip )
			suffix="tar.gz"
			tar_compression_flag="--${compression}"
		;;
		
		bzip2 )
			suffix="tar.bz2"
			tar_compression_flag="--${compression}"
		;;
		
		* )
			echo "Compression type ${compression} not implemented."
			e_fail 1
		;;
		
	esac
	
	archfile="`echo ${mountpoint} | sed -e\"s%/%_%g\"`"
	archive="${archdir}/${archfile}.${suffix}"
	if [ "${fstype}" == "xfs" ] && [ "${#mountpoint}" -gt "12" ]
	then
		# xfs labels are max. 12 chars long.
		label="$(basename \"${mountpoint}\")"
	else
		# prefix of /mnt is stripped for use from live CD or initramfs
		label="`echo ${mountpoint} | sed -e\"s%/mnt%/%\" | sed -e\"s%//%/%\"`"
		
	fi
	
	echo -e "Mountpoint to archivate:\t${mountpoint}"
	echo -e "Device to re-format:\t\t${device}"
	echo -e "Compression type:\t\t${compression}"
	echo -e "Archive file:\t\t\t${archive}"
	echo -e "New filesystem type:\t\t${fstype}"
	echo -e "Filesystem label:\t\t${label}"
	
	e_report "Re-mounting device readonly"
	mount -o remount,ro,noatime,nodiratime "${device}" "${mountpoint}" && e_done || e_fail 2
	
	e_report "Preparing archive directory"
	mkdir -p "${archdir}" &&
	touch "${archive}" && e_done || e_fail 2
	
	e_report "Creating archive"
	tar --create \
		--file "${archive}" \
		${tar_compression_flag} \
		--sparse \
		--one-file-system \
		--preserve-permissions \
		--atime-preserve \
		--absolute-names ${mountpoint} && e_done || e_fail 4
	
	e_report "Un-mounting the device"
	umount "${mountpoint}" && e_done || e_fail 2
	
#	e_report "Blanking the device"
#	dd if=/dev/zero of="${device}" bs=4M &> /dev/null
#	e_done

	e_report "Creating the filesystem"
	case ${fstype} in
	
		ext2 )
			mkfs.ext2 -q -O dir_index,filetype,sparse_super \
				-L "${label}" "${device}" && e_done || e_fail 8
		;;
		
		ext3 )
			mkfs.ext3 -q -O dir_index,filetype,sparse_super,has_journal -j \
				-L "${label}" "${device}" && e_done || e_fail 8
		;;
		
		reiserfs )
			mkfs.reiserfs -q --label="${label}" "${device}" >&/dev/null && e_done || e_fail 8
		;;
		
		reiser4 )
			mkfs.reiser4 -y --label="${label}" "${device}" >&/dev/null && e_done || e_fail 8
		;;
		
		xfs )
			mkfs.xfs -f -q -L "${label}" "${device}" >&/dev/null && e_done || e_fail 8
		;;
		
		* )
			echo "File system type ${fstype} not implemented."
			e_fail 1
		;;
		
	esac
	
	e_report "Mounting the new filesystem"
	mount -t ${fstype} -o rw,noatime,nodiratime "${device}" "${mountpoint}" && e_done || e_fail 2
	
	e_report "Unpacking data again"
	tar --extract \
		--file "${archive}" \
		${tar_compression_flag} \
		--sparse \
		--preserve-permissions \
		--atime-preserve \
		--absolute-names && e_done || e_fail 16
	
	e_report "Making sure _everything_ gets committed to disk"
	mount -t ${fstype} -o remount,ro,noatime,nodiratime "${device}" "${mountpoint}" && 
	mount -t ${fstype} -o remount,rw,noatime,nodiratime "${device}" "${mountpoint}" && 
	e_done || e_fail 2
	
	e_report "Removing archive"
	rm "${archive}" && e_done || e_fail 4
	rmdir "${archdir}" > /dev/null 2>&1
fi

exit 0

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: portage tree (Was: Re: reiser4 status (correction))

[Hans Reiser]

Thanks Christian.  You can go ahead and add something to our wiki
pointing to it if you would like.....  This might help tide people over
until the repacker ships.

Hans

wiki entry (Was: Re: portage tree)

[Christian Trefzer]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Sat, Jul 22, 2006 at 11:50:24PM -0600, Hans Reiser wrote:
> Thanks Christian.  You can go ahead and add something to our wiki
> pointing to it if you would like.....  This might help tide people
> over until the repacker ships.

I'd love to, but alas, wiki.namesys.com appears to have a serious
problem at the moment. The machine (193.232.112.68) responds to ping,
the http port is open but connections fail. The university's squid
proxies have quite some timeout, and I repeatedly get "connection reset
by peer". Google search result yields "This wiki is installed on a busy
underpowered server running Reiser4." which seems like a mild
understatement. How "busy" is that machine, actually?

Kind regards,
Chris

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: reiser4 status (correction)

[Hans Reiser]

Mike Benoit wrote:

>
>On top of that, I don't see how a repacker would help these work loads
>much as the files usually have a high churn rate. 
>
I think Reiserfs is used on a lot more than squid servers.  For them,
80% of files don't move for long periods of time is the usual industry
statistic....