ReiserFS slow, need help diagnosing

ReiserFS slow, need help diagnosing

[Juergen Starek]

Hello everyone,

I hope this is the right mailing list for my question, if not, please
redirect me.

I have noticed that if I put my ~ on a ReiserFS partition, Pan (GNOME's
newsreader) starts very slowly, taking about 75 seconds. If I move the
same article cache onto an ext3 partition, Pan starts in less than 10
seconds. This confuses me, as I've read a lot of benchmarks and comments
on the net which indicate that ReiserFS is faster than ext3 when handling
lots of small files.

Pan's article cache consists of lots of small files (approx. 70 MB, each
file has only a few kilobytes) and, on my system, it should not be
fragmented too much because I copied the files from an external source
onto the empty volume.

Asking this question on the German Linux newsgroup
de.comp.os.unix.linux.misc[1] didn't lead to any hints about diagnosing
this behaviour. Any ideas? 

Many thanks in advance

  Jürgen

[1]
http://groups.google.de/group/de.comp.os.unix.linux.misc/browse_thread/thread/d60014be503abebd/

Re: ReiserFS slow, need help diagnosing

[Vladimir V. Saveliev]

Hello

On Mon, 2006-06-05 at 15:11 +0200, Juergen Starek wrote:
> Hello everyone,
> 
> I hope this is the right mailing list for my question, if not, please
> redirect me.
> 
> I have noticed that if I put my ~ on a ReiserFS partition, Pan (GNOME's
> newsreader) starts very slowly, taking about 75 seconds. If I move the
> same article cache onto an ext3 partition, Pan starts in less than 10
> seconds. This confuses me, as I've read a lot of benchmarks and comments
> on the net which indicate that ReiserFS is faster than ext3 when handling
> lots of small files.
> 
> Pan's article cache consists of lots of small files (approx. 70 MB, each
> file has only a few kilobytes) and, on my system, it should not be
> fragmented too much because I copied the files from an external source
> onto the empty volume.
> 

Probably Pan is stat(2)-ing whole cache. Can you please try to find out
what Pan is doing on start? Strace(1) may help.
If Pan is doing millions of stats for files from cache - that explains
why it starts slowly when cache is stored on reiserfs.

> Asking this question on the German Linux newsgroup
> de.comp.os.unix.linux.misc[1] didn't lead to any hints about diagnosing
> this behaviour. Any ideas? 
> 
> Many thanks in advance
> 
>   Jürgen
> 
> [1]
> http://groups.google.de/group/de.comp.os.unix.linux.misc/browse_thread/thread/d60014be503abebd/
> 
> 

Re: ReiserFS slow, need help diagnosing

[Juergen Starek]

Vladimir V. Saveliev schrieb:
> On Mon, 2006-06-05 at 15:11 +0200, Juergen Starek wrote:
>
>> I have noticed that if I put my ~ on a ReiserFS partition, Pan (GNOME's
>> newsreader) starts very slowly, [...]
> 
> Probably Pan is stat(2)-ing whole cache. Can you please try to find out
> what Pan is doing on start? Strace(1) may help.

You're right, I straced Pan's start process and after initialization, it
seems to call stat64 on every message in its cache. This is a snippet of
the trace, produced on an ext3 filesystem (line length exceeds 80 chars): 

==========
stat64("/home/jstarek/.pan/gmane", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/jstarek/.pan/Standard", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

stat64("/home/jstarek/.pan/messages/cache", {st_mode=S_IFDIR|0755, st_size=1032192, ...}) = 0
open("/home/jstarek/.pan/messages/cache", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 6
fstat64(6, {st_mode=S_IFDIR|0755, st_size=1032192, ...}) = 0
fcntl64(6, F_SETFD, FD_CLOEXEC)         = 0
getdents64(6, /* 62 entries */, 4096)   = 4096
stat64("/home/jstarek/.pan/messages/cache/42b4e3b2$0$1128$9b4e6d93@newsread4.arcor-online.net.msg", {st_mode=S_IFREG|0644, st_size=1426, ...}) = 0
time(NULL)                              = 1149517138
==========

The last two calls of this trace snippet are then repeated for all files
in the cache.

> If Pan is doing millions of stats for files from cache - that explains
> why it starts slowly when cache is stored on reiserfs.

Could you please explain this? An URL with the explanation is fine, too --
I just don't have experience with file systems...

Thanks,

  Jürgen

Re: ReiserFS slow, need help diagnosing

[Vladimir V. Saveliev]

Hello

On Mon, 2006-06-05 at 16:29 +0200, Juergen Starek wrote:
> Vladimir V. Saveliev schrieb:
> > On Mon, 2006-06-05 at 15:11 +0200, Juergen Starek wrote:
> >
> >> I have noticed that if I put my ~ on a ReiserFS partition, Pan (GNOME's
> >> newsreader) starts very slowly, [...]
> > 
> > Probably Pan is stat(2)-ing whole cache. Can you please try to find out
> > what Pan is doing on start? Strace(1) may help.
> 
> You're right, I straced Pan's start process and after initialization, it
> seems to call stat64 on every message in its cache. This is a snippet of
> the trace, produced on an ext3 filesystem (line length exceeds 80 chars): 
> 
> ==========
> stat64("/home/jstarek/.pan/gmane", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> stat64("/home/jstarek/.pan/Standard", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> 
> stat64("/home/jstarek/.pan/messages/cache", {st_mode=S_IFDIR|0755, st_size=1032192, ...}) = 0
> open("/home/jstarek/.pan/messages/cache", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 6
> fstat64(6, {st_mode=S_IFDIR|0755, st_size=1032192, ...}) = 0
> fcntl64(6, F_SETFD, FD_CLOEXEC)         = 0
> getdents64(6, /* 62 entries */, 4096)   = 4096
> stat64("/home/jstarek/.pan/messages/cache/42b4e3b2$0$1128$9b4e6d93@newsread4.arcor-online.net.msg", {st_mode=S_IFREG|0644, st_size=1426, ...}) = 0
> time(NULL)                              = 1149517138
> ==========
> 
> The last two calls of this trace snippet are then repeated for all files
> in the cache.
> 
> > If Pan is doing millions of stats for files from cache - that explains
> > why it starts slowly when cache is stored on reiserfs.
> 
> Could you please explain this? An URL with the explanation is fine, too --
> I just don't have experience with file systems...
> 

reiserfs does not store inodes as compact as ext3. It allocates inodes
dynamically. As result reiserfs inodes get spread over whole filesystem.
Also reiserfs tries to store file bodies in the same block as file's
inode. As result one reiserfs block usually contains inodes than ext[23]
inode block does.
So, to stat each file of directory reiserfs has to perform more disk
reads and to do more disk head seek than a filesystem which stores
inodes compactly in preallocated disk area. 

> Thanks,
> 
>   Jürgen
> 
> 

Re: ReiserFS slow, need help diagnosing

[Grzegorz Kulewski]

On Mon, 5 Jun 2006, Vladimir V. Saveliev wrote:
> reiserfs does not store inodes as compact as ext3. It allocates inodes
> dynamically. As result reiserfs inodes get spread over whole filesystem.
> Also reiserfs tries to store file bodies in the same block as file's
> inode. As result one reiserfs block usually contains inodes than ext[23]
> inode block does.
> So, to stat each file of directory reiserfs has to perform more disk
> reads and to do more disk head seek than a filesystem which stores
> inodes compactly in preallocated disk area.

Is this performance problem with stat heavy load fixed in Reiser4?


Thanks,

Grzegorz Kulewski

Re: ReiserFS slow, need help diagnosing

[Vladimir V. Saveliev]

Hello

On Mon, 2006-06-05 at 17:34 +0200, Grzegorz Kulewski wrote:
> On Mon, 5 Jun 2006, Vladimir V. Saveliev wrote:
> > reiserfs does not store inodes as compact as ext3. It allocates inodes
> > dynamically. As result reiserfs inodes get spread over whole filesystem.
> > Also reiserfs tries to store file bodies in the same block as file's
> > inode. As result one reiserfs block usually contains inodes than ext[23]
> > inode block does.
> > So, to stat each file of directory reiserfs has to perform more disk
> > reads and to do more disk head seek than a filesystem which stores
> > inodes compactly in preallocated disk area.
> 
> Is this performance problem with stat heavy load fixed in Reiser4?
> 

Well, inode location in reiser4 changed comparing to reiserfs. reiser4
groups inodes of files of one directory together (reiserfs did not do
that), but still allocated disk space for inodes dynamically as
reiserfs.
So, I guess that reiser4 will be better than reiserfs, but 
still worse than ext[23]. Would you verify this guess it please?

> 
> Thanks,
> 
> Grzegorz Kulewski
> 
> 

Re: ReiserFS slow, need help diagnosing

[Grzegorz Kulewski]

On Mon, 5 Jun 2006, Vladimir V. Saveliev wrote:

> Hello
>
> On Mon, 2006-06-05 at 17:34 +0200, Grzegorz Kulewski wrote:
>> On Mon, 5 Jun 2006, Vladimir V. Saveliev wrote:
>>> reiserfs does not store inodes as compact as ext3. It allocates inodes
>>> dynamically. As result reiserfs inodes get spread over whole filesystem.
>>> Also reiserfs tries to store file bodies in the same block as file's
>>> inode. As result one reiserfs block usually contains inodes than ext[23]
>>> inode block does.
>>> So, to stat each file of directory reiserfs has to perform more disk
>>> reads and to do more disk head seek than a filesystem which stores
>>> inodes compactly in preallocated disk area.
>>
>> Is this performance problem with stat heavy load fixed in Reiser4?
>>
>
> Well, inode location in reiser4 changed comparing to reiserfs. reiser4
> groups inodes of files of one directory together (reiserfs did not do
> that), but still allocated disk space for inodes dynamically as
> reiserfs.
> So, I guess that reiser4 will be better than reiserfs, but
> still worse than ext[23]. Would you verify this guess it please?

I currently have no Reiser4 filesystem and no disk space to make one due 
to two recent big disk hardware failures. But I will buy some disks in 
about two months and I will try to experiment with Reiser4 again.

I (still) hope that Reiser4 will be included in mainline (more or less) 
soon...


Thanks,

Grzegorz Kulewski

Re: ReiserFS slow, need help diagnosing

[Hans Reiser]

Grzegorz Kulewski wrote:

>
>>
>> Well, inode location in reiser4 changed comparing to reiserfs. reiser4
>> groups inodes of files of one directory together (reiserfs did not do
>> that), but still allocated disk space for inodes dynamically as
>> reiserfs.
>> So, I guess that reiser4 will be better than reiserfs, but
>> still worse than ext[23]. Would you verify this guess it please?
>
I wouild not assume this.  There is a huge difference with respect to
this usage pattern between reiser4 and reiser3, it should dramatically
improve.  I don't know if we will be better or worse than ext3, it could
be either, best to measure it.

Re: ReiserFS slow, need help diagnosing

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Mon, 2006-06-05 at 09:22 -0700, Hans Reiser wrote:
> Grzegorz Kulewski wrote:
> 
> >
> >>
> >> Well, inode location in reiser4 changed comparing to reiserfs. reiser4
> >> groups inodes of files of one directory together (reiserfs did not do
> >> that), but still allocated disk space for inodes dynamically as
> >> reiserfs.
> >> So, I guess that reiser4 will be better than reiserfs, but
> >> still worse than ext[23]. Would you verify this guess it please?
> >
> I wouild not assume this.  There is a huge difference with respect to
> this usage pattern between reiser4 and reiser3, it should dramatically
> improve.  I don't know if we will be better or worse than ext3, it could
> be either, best to measure it.

I also use Pan, and recently switched to using Reiser4 on my laptop.  I
can tell that Pan starts much more quickly than it used to using
Reiser3.  It isn't nearly as fast as starting it from a USB Flash drive
though.  I don't know how it compares to ext3, although the flash drive
was using a ext2 loopback on FAT-32 filesystem.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: ReiserFS slow, need help diagnosing

[Juergen Starek]

Hans Reiser schrieb:

>>> So, I guess that reiser4 will be better than reiserfs, but
>>> still worse than ext[23]. Would you verify this guess it please?
>>
> I wouild not assume this.  There is a huge difference with respect to
> this usage pattern between reiser4 and reiser3, it should dramatically
> improve.  I don't know if we will be better or worse than ext3, it could
> be either, best to measure it.

Well, thanks to all of you for helping me with this. Once I get my hands
on a test system (and have enough free time), I'll try to test various
file systems with this scenario.

Regards,

  Jürgen

Re: ReiserFS slow, need help diagnosing

[Hans Reiser]

Vladimir V. Saveliev wrote:

>
> As result one reiserfs block usually contains 
>
fewer

>inodes than ext[23]
>inode block does.
>So, to stat each file of directory reiserfs has to perform more disk
>reads and to do more disk head seek than a filesystem which stores
>inodes compactly in preallocated disk area. 
>
>  
>
>>Thanks,
>>
>>  Jürgen
>>
>>
>>    
>>
>
>
>
>  
>