Silly question, defrag

Silly question, defrag

[Matthew Johnson]

This is kind of a general silly question, but one that crops now and again. 
Especially from newbies...

Whats the best, most accurate answer to give to a newbie when they ask how to 
defrag their hard drive, and does ReiserFS vary in itself with regards to 
this, with say ext2? Its just a question I sometimes get and wondered the 
best answer to this.

Kind regards,

Matt

Re: Silly question, defrag

[Joe Cooper]

"Don't"

;-)

ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth 
fighting over on most systems (certain environments are impacted more 
than others--mail servers and web caches being two examples that are hit 
pretty hard by fragmentation performance degradation).

Besides, there is no method to defrag ReiserFS that I know of.  Hans 
plans repacking in some future version.  It will be nice, but the whole 
'defrag once a month to keep your computer running smoothly' is kind of 
a Windows thing.  Us Unix users don't really need to think so much on 
those sorts of things.

Matthew Johnson wrote:
> This is kind of a general silly question, but one that crops now and again. 
> Especially from newbies...
> 
> Whats the best, most accurate answer to give to a newbie when they ask how to 
> defrag their hard drive, and does ReiserFS vary in itself with regards to 
> this, with say ext2? Its just a question I sometimes get and wondered the 
> best answer to this.
> 
> Kind regards,
> 
> Matt

-- 
Joe Cooper <joe@swelltech.com>
http://www.swelltech.com
Web Caching Appliances and Support

Re: Silly question, defrag

[Hans Reiser]

Joe Cooper wrote:

> "Don't"
>
> ;-)
>
> ReiserFS (and ext2|3) do fragment somewhat, but the impact is not 
> worth fighting over on most systems (certain environments are impacted 
> more than others--mail servers and web caches being two examples that 
> are hit pretty hard by fragmentation performance degradation).
>
> Besides, there is no method to defrag ReiserFS that I know of.  Hans 
> plans repacking in some future version.  It will be nice, but the 
> whole 'defrag once a month to keep your computer running smoothly' is 
> kind of a Windows thing.  Us Unix users don't really need to think so 
> much on those sorts of things. 

Defragging is a missing feature.  It will be the most important feature 
of V4.1

>
>
> Matthew Johnson wrote:
>
>> This is kind of a general silly question, but one that crops now and 
>> again. Especially from newbies...
>>
>> Whats the best, most accurate answer to give to a newbie when they 
>> ask how to defrag their hard drive, and does ReiserFS vary in itself 
>> with regards to this, with say ext2? Its just a question I sometimes 
>> get and wondered the best answer to this.
>>
>> Kind regards,
>>
>> Matt
>
>

Re: Silly question, defrag

[Matthew Johnson]

On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
> "Don't"
>

Well I don't, but when newbies who are used to computing on win32 systems 
hear that they may not just accept the word don't. Actually its hard to find 
the reasons exactly why one does not defrag.

> ;-)
>
> ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth
> fighting over on most systems (certain environments are impacted more
> than others--mail servers and web caches being two examples that are hit
> pretty hard by fragmentation performance degradation).
>
> Besides, there is no method to defrag ReiserFS that I know of.  Hans
> plans repacking in some future version.  It will be nice, but the whole
> 'defrag once a month to keep your computer running smoothly' is kind of
> a Windows thing.  Us Unix users don't really need to think so much on
> those sorts of things.

Perhaps I should aim this message to the kernel mailing list, so that I can 
get response from a wider array of people who like other filesystems. But its 
not kernel related. 

All this because of a simple query someone posted to a user group mailing 
list lol.

Matt

Re: Silly question, defrag

[Matthew Johnson]

> Defragging is a missing feature.  It will be the most important feature
> of V4.1

4.1 seems pretty exciting, but why the need to defrag on these filesystems vs 
others. Or do we not yet have the data? Guess I am intrigued on how these 
work. Will defrag be included for peace of mind, or for real gain? How much 
fragmentation can one expect? Will you need to unmount drives to defrag?

Sorry for all the questions...

Matt

Re: Silly question, defrag

[Hans Reiser]

Matthew Johnson wrote:

>On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
>
>>"Don't"
>>
>
>Well I don't, but when newbies who are used to computing on win32 systems 
>hear that they may not just accept the word don't. Actually its hard to find 
>the reasons exactly why one does not defrag.
>
>>;-)
>>
>>ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth
>>fighting over on most systems (certain environments are impacted more
>>than others--mail servers and web caches being two examples that are hit
>>pretty hard by fragmentation performance degradation).
>>
>>Besides, there is no method to defrag ReiserFS that I know of.  Hans
>>plans repacking in some future version.  It will be nice, but the whole
>>'defrag once a month to keep your computer running smoothly' is kind of
>>a Windows thing.  Us Unix users don't really need to think so much on
>>those sorts of things.
>>
>
>Perhaps I should aim this message to the kernel mailing list, so that I can 
>get response from a wider array of people who like other filesystems. But its 
>not kernel related. 
>
>All this because of a simple query someone posted to a user group mailing 
>list lol.
>
>Matt
>
>
There are ways in which Windows is better than Linux.  This is one.  To 
think that Linux is better for not having defrag is wishful innacurate 
thinking.  Such is life.  Send us $30k and defrag will go into v4.0 
instead of 4.1.:)

Hans

Re: Silly question, defrag

[Hans Reiser]

Matthew Johnson wrote:

>>Defragging is a missing feature.  It will be the most important feature
>>of V4.1
>>
>
>4.1 seems pretty exciting, but why the need to defrag on these filesystems vs 
>others. Or do we not yet have the data? Guess I am intrigued on how these 
>work. Will defrag be included for peace of mind, or for real gain? How much 
>fragmentation can one expect? Will you need to unmount drives to defrag?
>
>Sorry for all the questions...
>
>Matt
>
>
>
The reiser4.1 repacker will not just defrag, it will repack, and maybe 
even someday compress unaccessed data.

There are a whole host of optimizations that are too expensive to do 
with every write, but very reasonable to do once a day.  Repack will be 
great!

As for why is it not in V3, the reason is simple.  Code freeze hit, and 
it was not yet started....

Hans

Re: Silly question, defrag

[Anders Widman]

> "Don't"

> ;-)

> ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth 
> fighting over on most systems (certain environments are impacted more 
> than others--mail servers and web caches being two examples that are hit 
> pretty hard by fragmentation performance degradation).

> Besides, there is no method to defrag ReiserFS that I know of.  Hans 
> plans repacking in some future version.  It will be nice, but the whole 
> 'defrag once a month to keep your computer running smoothly' is kind of 
> a Windows thing.  Us Unix users don't really need to think so much on 
> those sorts of things.

Fragmentation is a problem with all filesystems. There is generally no
way around fragmentation other than "defragment".

If you want to add/store a large file on a 30% full filesystem it
would probably be stored on the first contingous area of free space.
This works fine until you have used most of the space and changed the
sizes of lots of files.

Fragmentation is enevitable when you only have small contingous blocks
of free/unallocated space and want to add a larger file. After some
time you end up with heavily fragmentation on any filesystem. Of
course, this doesn't happen when you don't add or change files.

//Anders


> Matthew Johnson wrote:
>> This is kind of a general silly question, but one that crops now and again. 
>> Especially from newbies...
>> 
>> Whats the best, most accurate answer to give to a newbie when they ask how to 
>> defrag their hard drive, and does ReiserFS vary in itself with regards to 
>> this, with say ext2? Its just a question I sometimes get and wondered the 
>> best answer to this.
>> 
>> Kind regards,
>> 
>> Matt

Re: Silly question, defrag

[Yura Umanets]

Anders Widman wrote:

>>"Don't"
>>
>
>>;-)
>>
>
>>ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth 
>>fighting over on most systems (certain environments are impacted more 
>>than others--mail servers and web caches being two examples that are hit 
>>pretty hard by fragmentation performance degradation).
>>
>
>>Besides, there is no method to defrag ReiserFS that I know of.  Hans 
>>plans repacking in some future version.  It will be nice, but the whole 
>>'defrag once a month to keep your computer running smoothly' is kind of 
>>a Windows thing.  Us Unix users don't really need to think so much on 
>>those sorts of things.
>>
>
>Fragmentation is a problem with all filesystems. There is generally no
>way around fragmentation other than "defragment".
>
>If you want to add/store a large file on a 30% full filesystem it
>would probably be stored on the first contingous area of free space.
>This works fine until you have used most of the space and changed the
>sizes of lots of files.
>
>Fragmentation is enevitable when you only have small contingous blocks
>of free/unallocated space and want to add a larger file. After some
>time you end up with heavily fragmentation on any filesystem. Of
>course, this doesn't happen when you don't add or change files.
>
>//Anders
>
This is one of things (fragmentation) about which people say "well known 
problem". Can you offer some solution?

-- 
Yury Umanets
IT Engineer of Priocom Corp.
Phone: +380 44 2011959, ICQ: 55494590

Re: Silly question, defrag

[matthew johnson]

> There are ways in which Windows is better than Linux.  This is one.  To
> think that Linux is better for not having defrag is wishful innacurate
> thinking.  Such is life.  Send us $30k and defrag will go into v4.0
> instead of 4.1.:)
>

Thanks for the answers, obviously 4.1 will be good thing. As for the cash,
well *cough* I can defiantely wait :). Although I imagine that filesystems
fragment differently depending on a number of factors (and that others are
more prone to more fragmentation...). Does ReiserFS deal with
fragmentation in any special way to minimise (ack, in USA thats minimize)
it?

Again, thanks so much!

Matt

Re: Silly question, defrag

[Ross Vandegrift]

On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote:
> On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
> > "Don't"
> >
> 
> Well I don't, but when newbies who are used to computing on win32 systems 
> hear that they may not just accept the word don't. Actually its hard to find 
> the reasons exactly why one does not defrag.

Depending on the elevator algorithms in Linux (sorry, I'm not very
familiar with them), performance can actually be *increased* by
some bit of fragmentation.  If the data is spread out over the disk,
I've heard it allows some elevator algorithms to improve their queuing
stategies.  I'm pretty sure this is the case on Novell Netware.

Unfortunately I don't have enough technical knowledge about elevator
algorithms to really know if it helps or hurts us.  However, I do know
that back in the days when I was just getting into Linux stuff
(around 2.0.0-ish era), it was generally said that ext2 driver did basic
defrag on write and it limited fragmentation to pathological filesystem
usage.

Again, not sure that this is still the case.

Ross Vandegrift
ross@willow.seitz.com

Re: Silly question, defrag

[Richard Thornton]

Why doesn't linux use UFS?  I can't understand this.

Richard


On Wed, 3 Apr 2002, Ross Vandegrift wrote:

> On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote:
> > On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
> > > "Don't"
> > >
> >
> > Well I don't, but when newbies who are used to computing on win32 systems
> > hear that they may not just accept the word don't. Actually its hard to find
> > the reasons exactly why one does not defrag.
>
> Depending on the elevator algorithms in Linux (sorry, I'm not very
> familiar with them), performance can actually be *increased* by
> some bit of fragmentation.  If the data is spread out over the disk,
> I've heard it allows some elevator algorithms to improve their queuing
> stategies.  I'm pretty sure this is the case on Novell Netware.
>
> Unfortunately I don't have enough technical knowledge about elevator
> algorithms to really know if it helps or hurts us.  However, I do know
> that back in the days when I was just getting into Linux stuff
> (around 2.0.0-ish era), it was generally said that ext2 driver did basic
> defrag on write and it limited fragmentation to pathological filesystem
> usage.
>
> Again, not sure that this is still the case.
>
> Ross Vandegrift
> ross@willow.seitz.com
>

Re: Silly question, defrag

[Hans Reiser]

Richard Thornton wrote:

>Why doesn't linux use UFS?  I can't understand this.
>
>Richard
>
>
>On Wed, 3 Apr 2002, Ross Vandegrift wrote:
>
>>On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote:
>>
>>>On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
>>>
>>>>"Don't"
>>>>
>>>Well I don't, but when newbies who are used to computing on win32 systems
>>>hear that they may not just accept the word don't. Actually its hard to find
>>>the reasons exactly why one does not defrag.
>>>
>>Depending on the elevator algorithms in Linux (sorry, I'm not very
>>familiar with them), performance can actually be *increased* by
>>some bit of fragmentation.  If the data is spread out over the disk,
>>I've heard it allows some elevator algorithms to improve their queuing
>>stategies.  I'm pretty sure this is the case on Novell Netware.
>>
>>Unfortunately I don't have enough technical knowledge about elevator
>>algorithms to really know if it helps or hurts us.  However, I do know
>>that back in the days when I was just getting into Linux stuff
>>(around 2.0.0-ish era), it was generally said that ext2 driver did basic
>>defrag on write and it limited fragmentation to pathological filesystem
>>usage.
>>
>>Again, not sure that this is still the case.
>>
>>Ross Vandegrift
>>ross@willow.seitz.com
>>
>
>
>
UFS is slower than ext2 by a lot.

Hans

Re: Silly question, defrag

[Anders Widman]

> Anders Widman wrote:

>>>"Don't"
>>>
>>
>>>;-)
>>>
>>
>>>ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth 
>>>fighting over on most systems (certain environments are impacted more 
>>>than others--mail servers and web caches being two examples that are hit 
>>>pretty hard by fragmentation performance degradation).
>>>
>>
>>>Besides, there is no method to defrag ReiserFS that I know of.  Hans 
>>>plans repacking in some future version.  It will be nice, but the whole 
>>>'defrag once a month to keep your computer running smoothly' is kind of 
>>>a Windows thing.  Us Unix users don't really need to think so much on 
>>>those sorts of things.
>>>
>>
>>Fragmentation is a problem with all filesystems. There is generally no
>>way around fragmentation other than "defragment".
>>
>>If you want to add/store a large file on a 30% full filesystem it
>>would probably be stored on the first contingous area of free space.
>>This works fine until you have used most of the space and changed the
>>sizes of lots of files.
>>
>>Fragmentation is enevitable when you only have small contingous blocks
>>of free/unallocated space and want to add a larger file. After some
>>time you end up with heavily fragmentation on any filesystem. Of
>>course, this doesn't happen when you don't add or change files.
>>
>>//Anders
>>
> This is one of things (fragmentation) about which people say "well known 
> problem". Can you offer some solution?

Yes, this is a well known problem, but I don't like some peoples ignorance
when they state that "this filesystem doesnt need defragmentation, it
is 'smarter' than other filesystems". I have heard this for both
Windows NT and for Linux systems.

I can't offer a sollution for this problem, but the probably easiest
way would be a application for doing offline defragmentation. But the
nicest way would be an online defragmentation.

//Anders

Re: Silly question, defrag

[Tracy R Reed]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote:
> On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
> > "Don't"
> >
> 
> Well I don't, but when newbies who are used to computing on win32 systems 
> hear that they may not just accept the word don't. Actually its hard to find 
> the reasons exactly why one does not defrag.

It is more useful to look at why one DID defrag back in the bad ol' days
of DOS and Windows. IIRC, the FAT filesystem would scan through it's
equivalient of the free block list and start writing at the first free
block. If it wrote for a while and then there was other data in the way it
stop and go to the next free space. This way fragmentation was practically
guarenteed and it happened rapidly. Modern filesystems use much smarter
ways of laying out data on the disk so that fragmentation happens much
less often. Now you will almost certainly waste more time by defragmenting
than you would suffering whatever performance hit the little fragmentation
there is causes. I've been using Linux/Unix for 10 years and I have never
(not once!) defragged a filesystem.

> Perhaps I should aim this message to the kernel mailing list, so that I can 
> get response from a wider array of people who like other filesystems. But its 
> not kernel related. 

I wouldn't recommend doing that. The answer is pretty much the same
regardless of the filesystem. If it's a non-FAT fs you probably don't have
to worry about fragmentation.

-- 
Tracy Reed      http://www.ultraviolet.org

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: Silly question, defrag

[Tracy R Reed]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

On Wed, Apr 03, 2002 at 10:24:37PM +0400, Hans Reiser wrote:
> There are ways in which Windows is better than Linux.  This is one.  To 
> think that Linux is better for not having defrag is wishful innacurate 
> thinking.  Such is life.  Send us $30k and defrag will go into v4.0 
> instead of 4.1.:)

I have to wonder about your motives here, Hans. You are the one who stands
to gain by capitalizing on newbie Unix/Linux users misunderstanding of
filesystems based on their experience with DOS and Windows and here you
are promoting defrag as a feature which puts your FS above others. They
have learned to compulsively defrag their disks once a week and you are
looking to feed their addiction. I think reiserfs is really great and a
defrag/repacker will be nice but the above strikes me as a bit strange.
How long has ext2 been around as the stock Linux filesystem?  More than
long enough for people to have realized whether a defragger would be
useful. Yet I can't think of a single distribution (of Linux or Unix in
general) that comes with a defragger. Stephen Tweedie wrote one for it but
nobody bothers to use it or even to include it with their distro.

Please don't perpetuate the idea that good filesystems have/need a
defragger.

-- 
Tracy Reed      http://www.ultraviolet.org
"She moves in mysterious ways"

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: Silly question, defrag

[rod]

<snip>

> I have to wonder about your motives here, Hans. You are the one who stands
> to gain by capitalizing on newbie Unix/Linux users misunderstanding of
> filesystems based on their experience with DOS and Windows and here you
> are promoting defrag as a feature which puts your FS above others. They
> have learned to compulsively defrag their disks once a week and you are
> looking to feed their addiction. I think reiserfs is really great and a
> defrag/repacker will be nice but the above strikes me as a bit strange.
> How long has ext2 been around as the stock Linux filesystem?  More than
> long enough for people to have realized whether a defragger would be
> useful. Yet I can't think of a single distribution (of Linux or Unix in
> general) that comes with a defragger. Stephen Tweedie wrote one for it but
> nobody bothers to use it or even to include it with their distro.
> 
> Please don't perpetuate the idea that good filesystems have/need a
> defragger.
> 
> -- 
> Tracy Reed      http://www.ultraviolet.org
> "She moves in mysterious ways"
> 
I'd actually suggest you are one of the poeple perpetuating a 
fragmentation myth.

Fragmentation is an artifact of random access to disk media.  It is 
caused when the final size of a file is not known by the filesystem 
layer at write time and hence it is not possible to allocate the total 
space in one contiguous block. Log files and variable database files 
that grow over time are a classic example of this scenario. 

This leaves the the only option to defrag once the file has been 
completed OR at some user defined interval. Some fs may try to 
reduce the incidence of this but cant prevent it happening. 

The response that "some fs's that are better than DOS/WinXX and 
dont have this problem" is simply a cop out and serve to confuse the 
issue.  Its these "linux is better than winxx" statements without a 
matching technical explanation that discredit us in the eyes of others.

Certainly the behavior of DOS/WIN systems in operation (endless 
upgrades, driver installs etc) promote fragmentation however some 
linux deployments fall foul to the same problem albeit for a different 
reason. (We have been bitten by this).

Currently the only way to defrag an ext2 fs (or reiserfs) is a copy 
off, mke2fs and copy back cycle - hardly ideal.  BTW if you are 
interested in scripts that capable of totally framenting an ext2 
partition (as reported by chke2fs and demonstrated by timing a  
grep) and also a discussion of the problem - check the archives.

Hans has recognised a genuine (though not popular or talked about) 
need in some applications of the fs and is fixing it. Good on him.


Cheers
-Rod

Re: Silly question, defrag

[Manuel Krause]

On 04/04/2002 01:49 AM, Tracy R Reed wrote:

> On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote:
> 
>>On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
>>
>>>"Don't"
>>>
>>>
>>Well I don't, but when newbies who are used to computing on win32 systems 
>>hear that they may not just accept the word don't. Actually its hard to find 
>>the reasons exactly why one does not defrag.
>>
> 
> It is more useful to look at why one DID defrag back in the bad ol' days
> of DOS and Windows. IIRC, the FAT filesystem would scan through it's
> equivalient of the free block list and start writing at the first free
> block. If it wrote for a while and then there was other data in the way it
> stop and go to the next free space. This way fragmentation was practically
> guarenteed and it happened rapidly. Modern filesystems use much smarter
> ways of laying out data on the disk so that fragmentation happens much
> less often. Now you will almost certainly waste more time by defragmenting
> than you would suffering whatever performance hit the little fragmentation
> there is causes. I've been using Linux/Unix for 10 years and I have never
> (not once!) defragged a filesystem.
> 
> 
>>Perhaps I should aim this message to the kernel mailing list, so that I can 
>>get response from a wider array of people who like other filesystems. But its 
>>not kernel related. 
>>
> 
> I wouldn't recommend doing that. The answer is pretty much the same
> regardless of the filesystem. If it's a non-FAT fs you probably don't have
> to worry about fragmentation.
> 
> 

Yesss. I like defragmenters on my Win98 disks, as I really see a speedup 
after using them (e.g after new software installations OR a longer 
status quo, but the effect depends on the defragmenters configuration).

You describe how/why this makes sense on FAT FS.

When I backupped my ReiserFS partitions monthly I used to recreate the 
original FS if everything was o.k. and copy back the whole content. 
That's no server here, it's a standalone notebook. After that procedure 
I found some applications that worked faster and some that were slower 
than before. O.k. I may have only subjectively compared the load times 
of NS6 +32MB disk cache and SO5.2. They were different than before 
copying, but the sum didn't show any advantage.

So?: I don't really need a Defragmenter on v3.6 ReiserFS in FAT scales 
for my usage and I really don't need to recreate and copy-back.

Mmh, just wanted to add my experience,

best wishes,

Manuel

Re: Silly question, defrag

[The Amazing Dragon]

> From: rod@tpgi.com.au
> 
> <snip>
> 
> > I have to wonder about your motives here, Hans. You are the one who stands
> > to gain by capitalizing on newbie Unix/Linux users misunderstanding of
> > filesystems based on their experience with DOS and Windows and here you
> > are promoting defrag as a feature which puts your FS above others. They
> > have learned to compulsively defrag their disks once a week and you are
> > looking to feed their addiction. I think reiserfs is really great and a
> > defrag/repacker will be nice but the above strikes me as a bit strange.
> > How long has ext2 been around as the stock Linux filesystem?  More than
> > long enough for people to have realized whether a defragger would be
> > useful. Yet I can't think of a single distribution (of Linux or Unix in
> > general) that comes with a defragger. Stephen Tweedie wrote one for it but
> > nobody bothers to use it or even to include it with their distro.

> This leaves the the only option to defrag once the file has been 
> completed OR at some user defined interval. Some fs may try to 
> reduce the incidence of this but cant prevent it happening. 

True. Some of the measures that are taken can be quite effective.
Reserving 10% overhead which only root is allowed to write in is
extremely effective with SunOS's flavor of UFS.

> Currently the only way to defrag an ext2 fs (or reiserfs) is a copy 
> off, mke2fs and copy back cycle - hardly ideal.  BTW if you are 
> interested in scripts that capable of totally framenting an ext2 
> partition (as reported by chke2fs and demonstrated by timing a  
> grep) and also a discussion of the problem - check the archives.

Not true, there is a e2defrag out there. Now, e2defrag will defragment a
filesystem, however the problem is that it greatly promotes future
fragmentation (got to maybe 2% fragmentation after months of use, but
within a week of running e2defrag it hit 8% fragmentation). I must
theorize that it packs files tightly rather than using the headroom
leading to the effect seen.


--
|\__/|\__/|\______          --=> 8-) EHM <=--          ______/|\__/|\__/|
\    |    |       | EHeM@cs.pdx.edu      PGP 8881EF59 |       |    |    /
  \   \   | ______| -O #include <stddisclaimer.h> O-  |______ |   /   /
    \___\_|/82 04 A1 3C C7 B1 37 2A   E3 6E 84 DA 97 4C 40 E6\|_/___/

Re: Silly question, defrag

[Anders Widman]

> On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote:
>> On Wednesday 03 April 2002 00:21, Joe Cooper wrote:
>> > "Don't"
>> >
>> 
>> Well I don't, but when newbies who are used to computing on win32 systems 
>> hear that they may not just accept the word don't. Actually its hard to find 
>> the reasons exactly why one does not defrag.

> It is more useful to look at why one DID defrag back in the bad ol' days
> of DOS and Windows. IIRC, the FAT filesystem would scan through it's
> equivalient of the free block list and start writing at the first free
> block. If it wrote for a while and then there was other data in the way it
> stop and go to the next free space. This way fragmentation was practically
> guarenteed and it happened rapidly. Modern filesystems use much smarter
> ways of laying out data on the disk so that fragmentation happens much
> less often. Now you will almost certainly waste more time by defragmenting
> than you would suffering whatever performance hit the little fragmentation
> there is causes. I've been using Linux/Unix for 10 years and I have never
> (not once!) defragged a filesystem.

>> Perhaps I should aim this message to the kernel mailing list, so that I can 
>> get response from a wider array of people who like other filesystems. But its 
>> not kernel related. 

> I wouldn't recommend doing that. The answer is pretty much the same
> regardless of the filesystem. If it's a non-FAT fs you probably don't have
> to worry about fragmentation.

I do not agree. I run a fileserver with a 814GB filesystem using
ReiserFS (I have run NTFS and ext2/3 also). Modern filesystems might
be smarter in storing new files by not packing them tightly.

In my case that workes fine up to a certain percentage, after that ALL
new files are beeing fragmented due to the fact that there is only
small blocks of space between all files. I don't see any filesystem
that don't need defragmentation. Not in my case.

//Anders

Re: Silly question, defrag

[Tracy R Reed]

[-- Attachment #1: Type: text/plain, Size: 2379 bytes --]

On Thu, Apr 04, 2002 at 11:16:51AM +0200, Anders Widman wrote:
> I do not agree. I run a fileserver with a 814GB filesystem using
> ReiserFS (I have run NTFS and ext2/3 also). Modern filesystems might
> be smarter in storing new files by not packing them tightly.
> 
> In my case that workes fine up to a certain percentage, after that ALL
> new files are beeing fragmented due to the fact that there is only
> small blocks of space between all files. I don't see any filesystem
> that don't need defragmentation. Not in my case.

Yes, after a certain percentage you will start getting fragmentation. Will
you really notice the performance hit? Who knows. It depends on how much
and which files get fragmented. One solution is to not fill disks up to
more than 90%. That is what most people who have a need to worry about
such things do. I'm just saying that it isn't worth it. If you are at 90%
you need to buy more disk anyway because soon you will be at 100%. 

If there were real value in regularly defragging then Veritas, Sun, IBM,
HP, and all of those guys would have made defraggers for their respective
filesystems and it would be considered best practice and standard
operating procedure to use them. But I have never heard of any such tools
nor procedures.

This whole discussion results from the fact that so many Linux people come
from PC backgrounds where they were taught to habitually defrag. People
who come from other systems never give it a thought.

Nonetheless, I look forward to having this functionality is reiserfs
because it certainly can't hurt. What interests me even more than
defragging is performance optimized layouts. If the filesystem can somehow
keep track of patterns of frequently accessed blocks and could recognize
that one set of blocks on the inner cylinders is always read immediately
after reading a set of blocks on the outer cylinder (or perhaps instead of
keeping track of blocks which are read it would be more efficient to keep
track of commonly performed long seeks and move data to remove those) and
could rearrange things so that all of the needed data passes under the
read head in most often used sequence we would see a MUCH bigger
improvement in performance.

-- 
Tracy Reed      http://www.ultraviolet.org
Would you buy a car with the hood welded shut?
Linux: the maintainable OS.

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: Silly question, defrag

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

On Thu, 04 Apr 2002 10:54:40 PST, Tracy R Reed <treed@ultraviolet.org>  said:

> If there were real value in regularly defragging then Veritas, Sun, IBM,
> HP, and all of those guys would have made defraggers for their respective
> filesystems and it would be considered best practice and standard
> operating procedure to use them. But I have never heard of any such tools
> nor procedures.

Actually, IBM *does* ship a 'defragfs' with its JFS file system under AIX.
When using file system compression under JFS, you can get horribly fragmented
under certain usage patterns.  'defragfs' is perfectly happy with running
on a live mounted filesystem.

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Silly question, defrag

[Anders Widman]

> On Thu, Apr 04, 2002 at 11:16:51AM +0200, Anders Widman wrote:
>> I do not agree. I run a fileserver with a 814GB filesystem using
>> ReiserFS (I have run NTFS and ext2/3 also). Modern filesystems might
>> be smarter in storing new files by not packing them tightly.
>> 
>> In my case that workes fine up to a certain percentage, after that ALL
>> new files are beeing fragmented due to the fact that there is only
>> small blocks of space between all files. I don't see any filesystem
>> that don't need defragmentation. Not in my case.

> Yes, after a certain percentage you will start getting fragmentation. Will
> you really notice the performance hit? Who knows. It depends on how much
> and which files get fragmented. One solution is to not fill disks up to
> more than 90%. That is what most people who have a need to worry about
> such things do. I'm just saying that it isn't worth it. If you are at 90%
> you need to buy more disk anyway because soon you will be at 100%. 

Yes, The perfomance hit is very noticeable. Also, consiber a system
with high uptime and many reads an writes (like my fileserver). Even
though a filsystem is better than other it only takes a little longer
before you see big framentation.

> If there were real value in regularly defragging then Veritas, Sun, IBM,
> HP, and all of those guys would have made defraggers for their respective
> filesystems and it would be considered best practice and standard
> operating procedure to use them. But I have never heard of any such tools
> nor procedures.

Isn't Microsoft considered a large company? Actually, Microsoft
said/stated that NTFS never needed defragmentation when Windows NT
came. It (Microsoft) was still holding on to this 'fact' with Windows
NT 4.0. The company, however, did change its policy with Windows 2000
and Windows XP.


> This whole discussion results from the fact that so many Linux people come
> from PC backgrounds where they were taught to habitually defrag. People
> who come from other systems never give it a thought.

Are you sure the opposite is not true as well? It would seem that they
(people from 'other' systems) are also thought "defrag is only for
windows".

> Nonetheless, I look forward to having this functionality is reiserfs
> because it certainly can't hurt. What interests me even more than
> defragging is performance optimized layouts. If the filesystem can somehow
> keep track of patterns of frequently accessed blocks and could recognize
> that one set of blocks on the inner cylinders is always read immediately
> after reading a set of blocks on the outer cylinder (or perhaps instead of
> keeping track of blocks which are read it would be more efficient to keep
> track of commonly performed long seeks and move data to remove those) and
> could rearrange things so that all of the needed data passes under the
> read head in most often used sequence we would see a MUCH bigger
> improvement in performance.

I agree. Preventing (minimizing) fragmentation is probably the best
choice, if not affecting write performance.

//Anders

Re: Silly question, defrag

[Rodd Zurcher]

Tracy R Reed wrote:

>If there were real value in regularly defragging then Veritas, Sun, IBM,
>HP, and all of those guys would have made defraggers for their respective
>filesystems and it would be considered best practice and standard
>operating procedure to use them. But I have never heard of any such tools
>nor procedures.
>
HP-UX's vxfs (veritas i believe) ships with a defrag tool that packs 
extents and reorders data based on access usage (dir order I believe). 
HPUX documentation did recommend running it and tuning how often by 
ammount of fragmentation. I found on a heavily used ClearCase server a 
nightly run kept fragmentation down; although a weekly run was probably 
all that was necessary. Haven't run HPUX in several years so things may 
be different now.

rbz

-- 
Rodd Zurcher
Principal Software Engineer
SPS/WBSG/WSAS - 847.576.0666
rodd.zurcher@motorola.com

Re: Silly question, defrag

[Benjamin Scott]

  It occurs to me that people are treating the answer to this question as a
single binary value.  I think that is an incorrect approach.

  By definition, any filesystem which is capable of storing files in
non-contiguous allocations can become fragmented.  The questions are, how
likely and how bad will it be?  It seems likely to me that different
filesystem implementations will have different behaviors with regards to
fragmentation, and those behaviors further vary depending on filesystem
usage patterns.

  So one question becomes, "Given a particular usage pattern, how fragmented
will a given filesystem become per unit of time?"  To use everyone's
favorite example, FAT's policy of simply allocating free blocks as it finds
them is practically worst-case for fragmentation.  I imagine it likely that
a more intelligent filesystem implementation would do better at this.

  Another question is, "Once a given filesystem becomes fragmented to a
given extent, what kind of performance impact do you see?"  This seems
likely to be an easier question to answer; you can do things like calculate
average seeks given various conditions, or just plain time a benchmark.  I
think it would seem likely that, once a filesystem does become fragmented,
performance is going to suck mud.  So most of it comes back to that first
question: Just how bad will the fragmentation get?

  I highly doubt this issue can be resolved to a single "yes or no" answer.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: Silly question, defrag

[rod]

<snip>

> When I started with Linux, I was told that Linux did not need a
> defragger because when saving a file, the file system (ext2 in this
> case) took the file size into account when looking for free blocks.
> Fragmentation only happens when the disk is almost full, and no
> contiguous free blocks can be found any more.
> 
This should true with files were the size is known before hand.  

> Windows, OTOH, starts writing at the first free block and just skips
> used blocks, which quickly leads to BIG fragmentation.
> 
> To defragment a disk, if I thought it was absolutely necessary, I was
> told to just read and re-write all files once. Of course for this to be
> successful, you need total free space in one block that exceeds the size
> of the biggest fragmented file.

Though tests I've done with copy & delete with partitions less than 
60% full showed that both ext2 and reiserfs did not correct 
fragmentation. My guess was that cp did the job in small chunks and 
the fs wasnt told the total to allocate.  This also demonstrated that for 
the case of multiple, continually growing files the amount of free 
space was irrelevant.

> 
> Are you telling me this is different with ReiserFS? Does ReiserFS not
> try to store files in one contiguous block if possible?
>  
> 
> -- 
> mfg, Jens Benecke  /// www.hitchhikers.de, www.linuxfaq.de, www.linux.ms
> This mail is an attachment? Read http://www.jensbenecke.de/misc/outlook.html
> 


----------------------------------------------
Rod Tunks
Hardware R&D Manager
TPG Network, Canberra
rod@tpgi.com.au  (02)62851711

"bad or missing Coffee.sys - operator halted"
ICQ: 4514607

Re: Silly question, defrag

[Ross Vandegrift]

> I think it would seem likely that, once a filesystem does become fragmented,
> performance is going to suck mud.

I'm curious though - shouldn't the elevator read algorithms take care of
this, for the most part?  If you wait to do a read until you have a few,
and read sequentially across the disk, you won't be hit with any seeks.

Of course interactive read performance will suffer, but for general
streaming reads why doesn't the elevator offset this?

Ross Vandegrift
ross@willow.seitz.com

Re: Silly question, defrag

[Anders Widman]

>> I think it would seem likely that, once a filesystem does become fragmented,
>> performance is going to suck mud.

> I'm curious though - shouldn't the elevator read algorithms take care of
> this, for the most part?  If you wait to do a read until you have a few,
> and read sequentially across the disk, you won't be hit with any seeks.

We should not forget that the kernel, or any software might not know
the physical layout of a physical disk medium. Also, if using
different raid, or spanning devices you have a logical layout for all
the blocks.

Read ahead and techniques like that do give lots of speedups, but not
so much for fragmentation.

//Anders


> Of course interactive read performance will suffer, but for general
> streaming reads why doesn't the elevator offset this?

> Ross Vandegrift
> ross@willow.seitz.com

Re: Silly question, defrag

[Hans Reiser]

There are a whole bunch of algorithms for optimizing layout for reads 
which are too expensive to do at every write.  Doing them once a week is 
an excellent way to handle them.

I am not well motivated as to talk about the advantages of features that 
ReiserFS lacks and other filesystems have..... but honesty requires it.

It is important to put airholes into the layout, so that insertions can 
go somewhere, and it would not surprise me if ext2's defrag failed to do 
that.

Hans

Re: Silly question, defrag

[Benjamin Scott]

On Fri, 5 Apr 2002, at 12:31am, Ross Vandegrift wrote:
>> I think it would seem likely that, once a filesystem does become fragmented,
>> performance is going to suck mud.
> 
> I'm curious though - shouldn't the elevator read algorithms take care of
> this, for the most part?  If you wait to do a read until you have a few,
> and read sequentially across the disk, you won't be hit with any seeks.

  I am no expert, but I suspect there is only so much the elevator can do.  
For a moderately small read, I image your scenario would work.  But given a
larger read and enough fragmentation, you are going to have to seek all over
the place.  Sure, you could increase the wait time in the elevator, and use
tons of buffers to re-order reads, but you will eventually start spending so
many resources on the elevator that you are back to a performance problem
again.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: Silly question, defrag

[Benjamin Scott]

On Fri, 5 Apr 2002, at 12:49pm, Hans Reiser wrote:
> There are a whole bunch of algorithms for optimizing layout for reads
> which are too expensive to do at every write.  Doing them once a week is
> an excellent way to handle them.

  That makes sense.

  What about using "idle time" to defragment the filesystem on a continuous
basis?  If the system is not doing anything else, a background daemon in the
kernel could move blocks around to optimize things.  While I expect this
would be a nightmare to implement in an existing filesystem driver, if it
were written with this in mind up-front, perhaps it would be easier?

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: Silly question, defrag

[Stefan Fleiter]

Hi Hans!

> It is important to put airholes into the layout, so that insertions can 
> go somewhere, and it would not surprise me if ext2's defrag failed to do 
> that.

Hum. Did I miss something?
Can you give a pointer to find out what you mean by »ext2's defrag«?

Greetings,
Stefan

Re: Silly question, defrag

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

On Fri, 05 Apr 2002 08:50:35 EST, Benjamin Scott <bscott@ntisys.com>  said:

>   What about using "idle time" to defragment the filesystem on a continuous
> basis?  If the system is not doing anything else, a background daemon in the
> kernel could move blocks around to optimize things.  While I expect this

A good idea, worth researching, but probably a royal pain to implement.
The biggest problem is identifying "idle", and being careful to do things
in a way that doesn't cause problems if the system suddenly goes un-idle.

It's not as bad as doing an fsck on a live filesystem - but you do want to
be sure that if a program suddenly decides to read the file you're currently
moving around, that you do the right thing (writing gets a bit trickier -
you have to worry about stuff like somebody doing an fsync() of a block
that you're in the middle of moving around on the disk.  Race Condition City ;)

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Silly question, defrag

[Benjamin Scott]

On Fri, 5 Apr 2002, at 9:04am, Valdis.Kletnieks@vt.edu wrote:
> The biggest problem is identifying "idle" ...

  This part of the algorithm could be farmed out to a userland system call.
That is, a user process signals the kernel when to start and stop
"background maintenance" tasks.  Said userland process could be as
sophisticated as something that monitors load average and I/O load, or as
simple as a cron job that runs during off-hours.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: Silly question, defrag

[Hans Reiser]

Benjamin Scott wrote:

>On Fri, 5 Apr 2002, at 12:31am, Ross Vandegrift wrote:
>
>>>I think it would seem likely that, once a filesystem does become fragmented,
>>>performance is going to suck mud.
>>>
>>I'm curious though - shouldn't the elevator read algorithms take care of
>>this, for the most part?  If you wait to do a read until you have a few,
>>and read sequentially across the disk, you won't be hit with any seeks.
>>
>
>  I am no expert, but I suspect there is only so much the elevator can do.  
>For a moderately small read, I image your scenario would work.  But given a
>larger read and enough fragmentation, you are going to have to seek all over
>the place.  Sure, you could increase the wait time in the elevator, and use
>tons of buffers to re-order reads, but you will eventually start spending so
>many resources on the elevator that you are back to a performance problem
>again.
>
Actually, the larger the read, the better the elevator can compensate. 
 The problem is that if you do a whole bunch of separate reads of a 
whole bunch of files, and you wait for each read to complete before 
doing the next one, the elevator cannot do a good job.  Also, if you 
have data you don't want to read interleaved with what you do want to 
read, then your transfer rate will be decreased.

Hans

Re: Silly question, defrag

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

>On Fri, 05 Apr 2002 08:50:35 EST, Benjamin Scott <bscott@ntisys.com>  said:
>
>>  What about using "idle time" to defragment the filesystem on a continuous
>>basis?  If the system is not doing anything else, a background daemon in the
>>kernel could move blocks around to optimize things.  While I expect this
>>
>
>A good idea, worth researching, but probably a royal pain to implement.
>The biggest problem is identifying "idle", and being careful to do things
>in a way that doesn't cause problems if the system suddenly goes un-idle.
>
>It's not as bad as doing an fsck on a live filesystem - but you do want to
>be sure that if a program suddenly decides to read the file you're currently
>moving around, that you do the right thing (writing gets a bit trickier -
>you have to worry about stuff like somebody doing an fsync() of a block
>that you're in the middle of moving around on the disk.  Race Condition City ;)
>
Actually, we will be able to do repacking online without a lot of extra 
code, due to our designing the right functionality into our flushing and 
journaling algorithms.

Some things become a lot easier when they are architected in at the 
beginning....

Hans

Re: Silly question, defrag

[Chris Dukes]

On Fri, Apr 05, 2002 at 03:53:09PM +0200, Stefan Fleiter wrote:
> Hi Hans!
> 
> > It is important to put airholes into the layout, so that insertions can 
> > go somewhere, and it would not surprise me if ext2's defrag failed to do 
> > that.
> 
> Hum. Did I miss something?
> Can you give a pointer to find out what you mean by »ext2's defrag«?

Next time could you try looking for 'defrag' on freshmeat?  Thanks.
http://freshmeat.net/projects/defrag/?topic_id=136 

It's also available as a debian package which is based off of the source
at http://www.go.dlr.de/linux/src/defrag-0.73.tar.gz.

A quick google search coughs up http://ftp.linux.org.uk/pub/linux/sct/defrag/

I suspect that the matter that sct hasn't seen fit to release a new version 
of defrag since 1998 has great meaning as to the relevance of defragging
under Linux.
-- 
Chris Dukes
"Bert is apparently EEEEVIL, whereas Oscar is just a sysadmin^Wgrouch."
-- gorski

Re: Silly question, defrag

[Stefan Fleiter]

Hi Chris!

On Fri, 05 Apr 2002 Chris Dukes wrote:

>> Can you give a pointer to find out what you mean by »ext2's defrag«?

> Next time could you try looking for 'defrag' on freshmeat?  Thanks.
> http://freshmeat.net/projects/defrag/?topic_id=136 

Sorry, I simply was to sure that something like this does not exist.
I had done this search when I started with Linux.
 
> I suspect that the matter that sct hasn't seen fit to release a new version 
> of defrag since 1998 has great meaning as to the relevance of defragging
> under Linux.

I don't think I would test this on any of my file systems, but ext2
is used solely for /usr over here. The rest resides on ReiserFS volumes.

But nice to know defrag for ext2 exists, though.

Thanks,
Stefan

Re: Silly question, defrag

[The Doctor What]

* Hans Reiser (reiser@namesys.com) [020403 12:27]:
> There are ways in which Windows is better than Linux.  This is one.  To 
> think that Linux is better for not having defrag is wishful innacurate 
> thinking.  Such is life.  Send us $30k and defrag will go into v4.0 
> instead of 4.1.:)

Sure, no problem:
#!/usr/bin/python

for i in range(0,30000):
    print "$",

 What else do you want? :-D

How many dollar signs do you want for ReiserFS under MacOS X?
*grin*

Ciao!

-- 
"There are monkey boys in the facility."
		--Yoyodyne Comm System (Adventures of Buckaroo Banzai)

The Doctor What: <fill in the blank>             http://docwhat.gerf.org/
docwhat@gerf.org                                                   KF6VNC