some testing questions

some testing questions

[Gasper Azman]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Hello everyone, especially the namesys team.


I have been reading this list for quite some time, and saw that issues
of fragmentation have again arisen. So, a decision was made to put the
portage tree on a separate reiser4 partition, and to benchmark it over time.

Because I know I'm not the smartest guy in the universe, I'm asking
everyone here which data they would require (or want to see), how to
obtain it (program names would suffice, but other advice will not go
amiss) and how frequently the analysis is to be run. Since this would be
a new partition with nothing but the portage tree, it is an ideal and
clean environment for testing a specific feature of reiser4 (as has been
frequently mentioned).

I hope to make a small contribution to the development of IMHO the most
promising filesystem since ext3. :)



Keep it up,


Gasper


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: some testing questions

[Vladimir V. Saveliev]

Hello

On Friday 11 August 2006 21:44, Gasper Azman wrote:
> Hello everyone, especially the namesys team.
>
>
> I have been reading this list for quite some time, and saw that issues
> of fragmentation have again arisen. So, a decision was made to put the
> portage tree on a separate reiser4 partition, and to benchmark it over
> time.
>
> Because I know I'm not the smartest guy in the universe, I'm asking
> everyone here which data they would require (or want to see), how to
> obtain it (program names would suffice, but other advice will not go
> amiss) and how frequently the analysis is to be run. 

reiser4progs incluses a program measurefs.reiser4. It should be able to 
measure tree fragmentation. I am not sure how does portage tree evolve, but 
maybe it could be interesting too see how does reiser4 tree fragmentation 
change when filesystem is loaded regularly.

> Since this would be 
> a new partition with nothing but the portage tree, it is an ideal and
> clean environment for testing a specific feature of reiser4 (as has been
> frequently mentioned).
>
> I hope to make a small contribution to the development of IMHO the most
> promising filesystem since ext3. :)
>
>
>
> Keep it up,
>
>
> Gasper

Re: some testing questions

[Ingo Bormuth]

On 2006-08-14 16:15, Vladimir V. Saveliev wrote:
> reiser4progs incluses a program measurefs.reiser4. It should be able to 
> measure tree fragmentation. I am not sure how does portage tree evolve, but 
> maybe it could be interesting too see how does reiser4 tree fragmentation 
> change when filesystem is loaded regularly.

This is a reiser4 partition holding the following:

  - portage tree (synced every three days)
  - ccache (compiler cache allowed to grow to 3GB - recently cleared)
  - firefox's and opera's cache
  - /tmp (portage builds everything in here)

The filesystem was created around 1.5 years ago (how can I say).

#cat /proc/version
Linux version 2.6.17.8-reiser4-r3 (root@kruemel) (gcc version 3.4.6 (Gentoo 3.4.6-r1, ssp-3.4.5-1.0, pie-8.7.9)) #2 Sat Aug 12 12:03:25 CEST 2006

#df:
/dev/hda8              6357768   3478716   2879052  55% /cache

#cat /etc/fstab:
/dev/hda8  /cache  reiser4  noatime,nodiratime,nodev,nosuid,tmgr.atom_max_age=500000  0 0

#measurefs.reiser4 -S:
measurefs.reiser4 1.0.5
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by
reiser4progs/COPYING.

Tree statistics ... done
Packing statistics:
  Formatted nodes:    3622.85b (88.45%)
  Branch nodes:       2792.00b (68.16%)
  Twig nodes:         3233.75b (78.95%)
  Leaf nodes:         3966.47b (96.84%)

Node statistics:
  Total nodes:         871653
  Formatted nodes:      75571
  Unformatted nodes:   796082
  Branch nodes:            23
  Twig nodes:            1360
  Leaf nodes:          870270

Item statistics:
  Total items:         542211
  Nodeptr items:        75570
  Statdata items:     214695
  Direntry items:       37432
  Tail items:          207819
  Extent items:          6695


Tree fragmentation: 0.074648

Data fragmentation: 0.039962

Last week I recompiled gcc and afterwards cleared 3GB of ccache data. 
Before doing so, the partition was >90% full. My feeling is that now 
that it's half empty performance is much better. Emerge sync used to 
take _ages_ rebuilding its cache and now is quite fast. Also CPU usage 
during compilation seems much lower. I can't remember to ever hear the 
CPU fan running during recent compilations (700MHZ PIII). Before clearing 
the cache it ran continuously and still felt hot.

I know none of this is hard data. If you are interested in a follow up,
just let me know.


BTW: Is it save to run measurefs.reiser4 -S -T -D on a mounted fs ?

-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517       '(~o-o~)'
public key 86326EC9, http://ibormuth.efil.de/contact   --ooO--(.)--Ooo--

Re: some testing questions

[Hans Reiser]

Ingo Bormuth wrote:
>
>
> #df:
> /dev/hda8              6357768   3478716   2879052  55% /cache
>  
> Before doing so, the partition was >90% full. 
The performance difference between >90% full and 55% full will be large
on every filesystem.  When we ship a repacker, that will be less true,
because we will have large chunks of unused space after the repacker runs.

Oddly enough, I don't know the statistics for reiser* filesystems, but I
know that for FFS you should not let it become more than 85% full before
buying a new disk (or cleaning your home directory) if you want good
performance.

Re: some testing questions

[David Masover]

Hans Reiser wrote:
> Ingo Bormuth wrote:
>>
>> #df:
>> /dev/hda8              6357768   3478716   2879052  55% /cache
>>  
>> Before doing so, the partition was >90% full. 
> The performance difference between >90% full and 55% full will be large
> on every filesystem.  When we ship a repacker, that will be less true,
> because we will have large chunks of unused space after the repacker runs.

Not always true.  For one, doesn't Reiser4 arbitrarily reserve 5%?  For 
another, look at his results -- unless I'm wrong, that's 3-7% 
fragmentation.  If I'm wrong, it's more like .03-.07%.

And lastly, at a certain point, percentages aren't really that accurate. 
  I've got a 350 or 400 gig partition which is 95% full, according to df 
(which if I was right about that 5%, it's more like 90% full) and that 
still leaves a solid 10-20 gigs free.

I mean, yes, performance will eventually start to suffer, but how much 
time and activity will it take to fragment 20 gigs of free space, 
especially with lazy allocation?

Re: some testing questions

[Hans Reiser]

David Masover wrote:
>  that's 3-7% fragmentation.  
which is high enough to hurt performance..... 50mb/s * 0.01 seconds =
amount of transfer a seek costs.  He needs a repacker.  After we resolve
code review issues from akpm.....

Re: some testing questions

[Gasper Azman]

[-- Attachment #1.1.1: Type: text/plain, Size: 1558 bytes --]

Vladimir V. Saveliev wrote:
> Hello
>
> On Friday 11 August 2006 21:44, Gasper Azman wrote:
>   
>> Hello everyone, especially the namesys team.
>>
>>
>> I have been reading this list for quite some time, and saw that issues
>> of fragmentation have again arisen. So, a decision was made to put the
>> portage tree on a separate reiser4 partition, and to benchmark it over
>> time.
>>
>> Because I know I'm not the smartest guy in the universe, I'm asking
>> everyone here which data they would require (or want to see), how to
>> obtain it (program names would suffice, but other advice will not go
>> amiss) and how frequently the analysis is to be run. 
>>     
>
> reiser4progs incluses a program measurefs.reiser4. It should be able to 
> measure tree fragmentation. I am not sure how does portage tree evolve, but 
> maybe it could be interesting too see how does reiser4 tree fragmentation 
> change when filesystem is loaded regularly.
>   
Ok. I'm gonna see about the program, and run it after each emerge --sync.

About the portage tree evolving, it's a lot (~150000) of <2k files, some
never change, and some change very often. New ones are added and old
ones deleted on a similar principle. People have been complaining for
quite some time that, while at first they experienced major speedups
regarding the operation of portage and the rsync process, it slowed to a
halt about half a year afterwards. Some blame fragmentation, and this
may be the key to diagnosing the problem.

Thanks for your help.

Gasper

[-- Attachment #1.1.2: Type: text/html, Size: 2061 bytes --]

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]