Re: More Slowdown

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

On Sat, 2005-11-12 at 09:06 +0000, John Gilmore wrote:
> Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
> soon. The specific symptom is that the effected process locks for a time, 
> usually just a second or two, but sometimes a minute or two and and at least 
> once for many many minutes. I think that the crash (soft lockup) that I 
> reported earlier is related as well. And it sounds like the comment that 
> rvalles had about lockups with mmaped files, except that it doesn't lock up 
> permanently. Just for a second or three usually.
> 

Hi,

I am sorry to say that I am having the same slowdown issue with Reiser4
and kernel 2.6.14.2. It seems that my problem is not related to the I/O
scheduler algorithm as I have tried both cfq and anticipatory.

The I/O activity seems to come in long bursts lasting up to 15 seconds,
and is unpredictable. The system almost grinds to a halt while the burst
of I/O is taking place, which can be quite annoying while typing.
There are no warnings in dmesg/syslog and the filesystem is clean.

Sometimes the burst of I/O activity occurs in a different form, with
short bursts every second, lasting about 10 seconds. Approx 30% duty
cycle. When a burst occurs in this form, it tends not to lock processes
as much.

The general sound of the drive when a burst is in progress is that of
heavy head movement, as if the drive is repeatedly seeking between two
or three locations.

Also, a thing which I noticed a while back, and may not be related in
any way to Reiser4.. When I re-size the columns in Evolution Mail, I get
continuous hard disk activity during the re-size. This hard disk
activity stops either when I release the mouse or hold it still with the
button down. Just wondering if anyone else gets the same thing.

Regards,
-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[jp]

Hi,

For me reiser4 is working perfectly on Vanilla 2.6.14.2, all the testers 
who I sent the new kernel to are reporting excellent performance and 
reliability.

Could you explain a bit further the way I could reproduce this ?

Thanks

JP

Craig Shelley wrote:

>On Sat, 2005-11-12 at 09:06 +0000, John Gilmore wrote:
>  
>
>>Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to 
>>soon. The specific symptom is that the effected process locks for a time, 
>>usually just a second or two, but sometimes a minute or two and and at least 
>>once for many many minutes. I think that the crash (soft lockup) that I 
>>reported earlier is related as well. And it sounds like the comment that 
>>rvalles had about lockups with mmaped files, except that it doesn't lock up 
>>permanently. Just for a second or three usually.
>>
>>    
>>
>
>Hi,
>
>I am sorry to say that I am having the same slowdown issue with Reiser4
>and kernel 2.6.14.2. It seems that my problem is not related to the I/O
>scheduler algorithm as I have tried both cfq and anticipatory.
>
>The I/O activity seems to come in long bursts lasting up to 15 seconds,
>and is unpredictable. The system almost grinds to a halt while the burst
>of I/O is taking place, which can be quite annoying while typing.
>There are no warnings in dmesg/syslog and the filesystem is clean.
>
>Sometimes the burst of I/O activity occurs in a different form, with
>short bursts every second, lasting about 10 seconds. Approx 30% duty
>cycle. When a burst occurs in this form, it tends not to lock processes
>as much.
>
>The general sound of the drive when a burst is in progress is that of
>heavy head movement, as if the drive is repeatedly seeking between two
>or three locations.
>
>Also, a thing which I noticed a while back, and may not be related in
>any way to Reiser4.. When I re-size the columns in Evolution Mail, I get
>continuous hard disk activity during the re-size. This hard disk
>activity stops either when I release the mouse or hold it still with the
>button down. Just wondering if anyone else gets the same thing.
>
>Regards,
>  
>

Re: More Slowdown

[Christian Iversen]

On Monday 14 November 2005 20:41, Craig Shelley wrote:
> On Sat, 2005-11-12 at 09:06 +0000, John Gilmore wrote:
> > Oh, BTW. "The slowdown" as I called it is still there. I guess I spoke to
> > soon. The specific symptom is that the effected process locks for a time,
> > usually just a second or two, but sometimes a minute or two and and at
> > least once for many many minutes. I think that the crash (soft lockup)
> > that I reported earlier is related as well. And it sounds like the
> > comment that rvalles had about lockups with mmaped files, except that it
> > doesn't lock up permanently. Just for a second or three usually.
>
> Hi,
>
> I am sorry to say that I am having the same slowdown issue with Reiser4
> and kernel 2.6.14.2. It seems that my problem is not related to the I/O
> scheduler algorithm as I have tried both cfq and anticipatory.
>
> The I/O activity seems to come in long bursts lasting up to 15 seconds,
> and is unpredictable. The system almost grinds to a halt while the burst
> of I/O is taking place, which can be quite annoying while typing.
> There are no warnings in dmesg/syslog and the filesystem is clean.
>
> Sometimes the burst of I/O activity occurs in a different form, with
> short bursts every second, lasting about 10 seconds. Approx 30% duty
> cycle. When a burst occurs in this form, it tends not to lock processes
> as much.
>
> The general sound of the drive when a burst is in progress is that of
> heavy head movement, as if the drive is repeatedly seeking between two
> or three locations.
>
> Also, a thing which I noticed a while back, and may not be related in
> any way to Reiser4.. When I re-size the columns in Evolution Mail, I get
> continuous hard disk activity during the re-size. This hard disk
> activity stops either when I release the mouse or hold it still with the
> button down. Just wondering if anyone else gets the same thing.

That's _serisouly_ odd. I've seen something like this happen before, when X 
programs generated some (non-harmful) X warnings. These were then written 
into a log file, typically .xsession-errors or similar. Any chance that's 
what you are seeing?

-- 
Regards,
Christian Iversen

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On Mon, 2005-11-14 at 21:47 +0100, Christian Iversen wrote:
> That's _serisouly_ odd. I've seen something like this happen before, when X 
> programs generated some (non-harmful) X warnings. These were then written 
> into a log file, typically .xsession-errors or similar. Any chance that's 
> what you are seeing?
> 

This explains the problem with Evolution, it looks a bit trigger happy
with the fsync call while resizing columns.
craig@teratron.lan.etheus.net:~$ strace -p 8002 2>&1 | grep fsync
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0
fsync(37)                               = 0

while true; do lsof  -p 8002 2>&1 | grep 37w ; done
evolution 8002 craig   37w   REG        3,4     417
1901436 /mnt/storage/craig-home/.evolution/mail/views/.#custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xmlwhile 

The file listed above seems to get moved, but is only a few hundred
bytes.

ls -lh /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox
\:_home_craig_.evolution_mail_local#Lists_Reiser4.xml
-rw-------  1 craig craig 417 2005-11-15
14:19 /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xml

Why does calling fsync on this one small file cause so much hard disk
activity?


-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[Laurent Riffard]

Le 15.11.2005 15:27, Craig Shelley a écrit :
> On Mon, 2005-11-14 at 21:47 +0100, Christian Iversen wrote:
> 
>>That's _serisouly_ odd. I've seen something like this happen before, when X 
>>programs generated some (non-harmful) X warnings. These were then written 
>>into a log file, typically .xsession-errors or similar. Any chance that's 
>>what you are seeing?
>>
> 
> This explains the problem with Evolution, it looks a bit trigger happy
> with the fsync call while resizing columns.
> craig@teratron.lan.etheus.net:~$ strace -p 8002 2>&1 | grep fsync
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0
> fsync(37)                               = 0

Please, could you do it again with the -T option for strace? It will
show the time spent in system  calls.

> while true; do lsof  -p 8002 2>&1 | grep 37w ; done
> evolution 8002 craig   37w   REG        3,4     417
> 1901436 /mnt/storage/craig-home/.evolution/mail/views/.#custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xmlwhile 
> 
> The file listed above seems to get moved, but is only a few hundred
> bytes.
> 
> ls -lh /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox
> \:_home_craig_.evolution_mail_local#Lists_Reiser4.xml
> -rw-------  1 craig craig 417 2005-11-15
> 14:19 /mnt/storage/craig-home/.evolution/mail/views/custom_view-mbox:_home_craig_.evolution_mail_local#Lists_Reiser4.xml
> 
> Why does calling fsync on this one small file cause so much hard disk
> activity?

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

On Tue, 2005-11-15 at 19:04 +0100, Laurent Riffard wrote:
> Please, could you do it again with the -T option for strace? It will
> show the time spent in system  calls.
> 
craig@teratron.lan.etheus.net:~$ strace -T -p 8002 2>&1 | grep fsync
fsync(26)                               = 0 <0.566159>
fsync(26)                               = 0 <0.159852>
fsync(26)                               = 0 <0.179659>
fsync(26)                               = 0 <0.153406>
fsync(26)                               = 0 <0.136685>
fsync(26)                               = 0 <0.171057>
fsync(26)                               = 0 <0.165733>
fsync(26)                               = 0 <0.178846>
fsync(26)                               = 0 <0.186595>
fsync(26)                               = 0 <0.184368>
fsync(26)                               = 0 <0.177273>
fsync(26)                               = 0 <0.178634>
fsync(26)                               = 0 <0.181036>
fsync(26)                               = 0 <0.178750>
fsync(26)                               = 0 <0.324480>
fsync(26)                               = 0 <0.177226>
fsync(26)                               = 0 <0.184625>
fsync(26)                               = 0 <0.186224>
fsync(26)                               = 0 <0.177282>
fsync(26)                               = 0 <0.177146>
fsync(26)                               = 0 <0.176000>
fsync(26)                               = 0 <0.192434>
fsync(26)                               = 0 <0.177482>
fsync(26)                               = 0 <0.183357>
fsync(26)                               = 0 <0.176528>
fsync(26)                               = 0 <0.184334>
fsync(26)                               = 0 <0.176622>
fsync(26)                               = 0 <0.177977>
fsync(26)                               = 0 <0.178636>
fsync(26)                               = 0 <0.259863>
fsync(26)                               = 0 <0.172789>
fsync(26)                               = 0 <0.171739>
fsync(26)                               = 0 <0.179641>
fsync(26)                               = 0 <0.178157>
fsync(26)                               = 0 <0.180272>
fsync(26)                               = 0 <0.170426>
fsync(26)                               = 0 <0.187630>
fsync(26)                               = 0 <0.171589>
fsync(26)                               = 0 <0.178135>


I seem to be finding that the performance of the system degrades with
time and file system usage.
Openoffice really starts to run slow after a while. Just as an
experiment, I did 
ls -R /usr
The directory listing appeared very quickly with hardly any hard disk
access. Then shortly later when switching back to Openoffice, the
process froze for about 3 minutes with continuous hard disk access.

One thing which may be relevant is that my kernel is patched with
Suspend2 and FBsplash, although I can't remember having any problems
like this before.

Many Thanks,

-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[Thorsten Hirsch]

Hi,

I've seen the mails in the mail archive concerning a slow-down problem
with reiser4. Seems like I've got the same problem. My kernel is
2.6.14-mm2 (gentoo) and my default disk scheduler is anticipatory, but
I've also tried cqf and I'd say that it depends on the application for
which scheduler might be a bis faster. But they're both very slow and my
load is very high.

I've found out, that resizing the columns in evolution is also for me a
good proof of this problem as it really creates a lot of hard disk
activity. So I've done the same strace call as Craig Shelley:

bauerbob ~ # echo anticipatory > /sys/block/hda/queue/scheduler
bauerbob ~ # strace -T -p 6336 2>&1 | grep fsync
fsync(52)                               = 0 <1.812899>
fsync(52)                               = 0 <0.734160>
fsync(52)                               = 0 <0.427722>
fsync(52)                               = 0 <0.839306>
fsync(52)                               = 0 <0.304873>
fsync(52)                               = 0 <0.236160>
fsync(60)                               = 0 <0.469446>
fsync(60)                               = 0 <0.460405>
fsync(60)                               = 0 <0.468343>
fsync(61)                               = 0 <0.745482>
fsync(61)                               = 0 <0.869358>
fsync(64)                               = 0 <1.500842>

bauerbob ~ # echo cfq > /sys/block/hda/queue/scheduler
bauerbob ~ # strace -T -p 6336 2>&1 | grep fsync
fsync(52)                               = 0 <3.131074>
fsync(52)                               = 0 <0.806179>
fsync(52)                               = 0 <0.341460>
fsync(52)                               = 0 <0.212601>
fsync(52)                               = 0 <0.251179>
fsync(52)                               = 0 <0.614537>
fsync(52)                               = 0 <0.246023>
fsync(52)                               = 0 <0.238064>
fsync(52)                               = 0 <0.386216>
fsync(52)                               = 0 <0.205596>
fsync(52)                               = 0 <0.250840>
fsync(52)                               = 0 <0.246623>
fsync(52)                               = 0 <0.242773>

PID 6336 is evolution.
Please let me know how I can help you.


Best regards,
Thorsten

Re: More Slowdown

[Artur Makówka]

Thorsten Hirsch wrote:
 > Hi,
 >
 > I've seen the mails in the mail archive concerning a slow-down problem
 > with reiser4. Seems like I've got the same problem. My kernel is
 > 2.6.14-mm2 (gentoo) and my default disk scheduler is anticipatory, but
 > I've also tried cqf and I'd say that it depends on the application for
 > which scheduler might be a bis faster. But they're both very slow and my
 > load is very high.
 >
 > I've found out, that resizing the columns in evolution is also for me a
 > good proof of this problem as it really creates a lot of hard disk
 > activity. So I've done the same strace call as Craig Shelley:
 >
[cut]
 >

My slowdown problems are on my web server, so its concerning different 
things than Evolution. but this slowdown was very easy to notice when 
writing something with vim.

Anyways, i remembered that i was using stable kernel without this 
happening some time ago. So i tested 2.6.14.x , 2.6.13.x mm and 2.6.13.x 
, and i still had system crashes. so i downgraded to 2.6.12.6 and the 
system is stable for more than 24 hours (first time in this month or 
even longer!)

probably some processes were doing fsync often (or something similar) 
and the effect was raising LA and lock-up at the end.

anyways, 2.6.12.6 (but other than .6 probably too) is, i think, bug 
free. maybe this will help.

Re: More Slowdown

[Andreas Rosander]

> My slowdown problems are on my web server, so its concerning different
> things than Evolution. but this slowdown was very easy to notice when
> writing something with vim.
>
> Anyways, i remembered that i was using stable kernel without this
> happening some time ago. So i tested 2.6.14.x , 2.6.13.x mm and 2.6.13.x
> , and i still had system crashes. so i downgraded to 2.6.12.6 and the
> system is stable for more than 24 hours (first time in this month or
> even longer!)
>
> probably some processes were doing fsync often (or something similar)
> and the effect was raising LA and lock-up at the end.
>
> anyways, 2.6.12.6 (but other than .6 probably too) is, i think, bug
> free. maybe this will help.
>
>

Well i do recon that I to have slow downs with vim or rather gvim and
I have noticed it when im use gedit too.
will try to downgrade too 2.6.12 and see if my box will rid of the slowdowns

//Andreas

Re: More Slowdown

[Avuton Olrich]

On 11/15/05, Andreas Rosander <andreasrosander@gmail.com> wrote:
> Well i do recon that I to have slow downs with vim or rather gvim and
> I have noticed it when im use gedit too.
> will try to downgrade too 2.6.12 and see if my box will rid of the slowdowns

It's funny that you mention vim. vim seems to be what _really_ makes
my reiser4 do the 'slowdown'. I call it harddrive thrashing cause
that's what my wife calls it when she hears it from 5 yards away :)
Right before saving or saving/exiting it really does this thrashing,
Thank god you said this because I didn't think this 'slowdown' was the
same thing I was experiencing.

avuton
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: More Slowdown

[Artur Makówka]

Avuton Olrich wrote:
> It's funny that you mention vim. vim seems to be what _really_ makes
> my reiser4 do the 'slowdown'. I call it harddrive thrashing cause
> that's what my wife calls it when she hears it from 5 yards away :)
> Right before saving or saving/exiting it really does this thrashing,
> Thank god you said this because I didn't think this 'slowdown' was the
> same thing I was experiencing.
> 
> avuton

i've mentioned vim becouse someone else already mention it too :) But i 
remember that with vim i had very long pauses.

also apt-get was slow. i don't even see the pattern, joe was also 
slowed, but not as much as vim. It's strange. does vim save its files in 
/tmp maybe ? or is there anything specific about vim that, for example, 
joe doesnt do? (and therefore much shorter slowdowns).

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Tue, 2005-11-15 at 15:13 -0800, Avuton Olrich wrote:
> It's funny that you mention vim. vim seems to be what _really_ makes
> my reiser4 do the 'slowdown'. I call it harddrive thrashing cause
> that's what my wife calls it when she hears it from 5 yards away :)
> Right before saving or saving/exiting it really does this thrashing,
> Thank god you said this because I didn't think this 'slowdown' was the
> same thing I was experiencing.

This looks to be a similar thing with fsync(). It took approx 10 sec to
save a file containing the string "Hello World" 
The results indicate that the time was spent in the fsync() call.

craig@teratron.lan.etheus.net:~$ ps aux | grep vim
craig     9111  0.0  0.5 17268 5936 pts/0    S+   23:35   0:00 vim
temp/testfile

craig@teratron.lan.etheus.net:~$ strace -T -p 9111 2>&1 | grep fsync
fsync(4)                                = 0 <9.726565>


-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[David Masover]

Craig Shelley wrote:
> On Tue, 2005-11-15 at 15:13 -0800, Avuton Olrich wrote:
> 
>>It's funny that you mention vim. vim seems to be what _really_ makes
>>my reiser4 do the 'slowdown'. I call it harddrive thrashing cause
>>that's what my wife calls it when she hears it from 5 yards away :)
>>Right before saving or saving/exiting it really does this thrashing,
>>Thank god you said this because I didn't think this 'slowdown' was the
>>same thing I was experiencing.
> 
> 
> This looks to be a similar thing with fsync(). It took approx 10 sec to
> save a file containing the string "Hello World" 
> The results indicate that the time was spent in the fsync() call.

I got sick of waiting for it and nuked the fsync call.  All my kernels
have a custom patch such that sys_fsync just returns true, no matter what.

Why?

Because vim shouldn't fsync, and neither should Evolution.  It's been so
abused that I prefer to just manually run "sync" when I want something
flushed.

Even if fysnc was fast (only flushing the file that needed to flush),
that kind of abuse -- resizing a column -- kind of kills any advantage
of lazy writes.

Re: More Slowdown

[Francesco Biscani]

On Wednesday 16 November 2005 01:45, David Masover wrote:
> I got sick of waiting for it and nuked the fsync call.  All my kernels
> have a custom patch such that sys_fsync just returns true, no matter what.

Mhh.. would it be something like this?

--- buffer.c.old        2005-11-16 02:36:46.129829994 +0100
+++ buffer.c    2005-11-16 02:37:11.125079752 +0100
@@ -376,7 +376,7 @@

 asmlinkage long sys_fsync(unsigned int fd)
 {
-       return do_fsync(fd, 0);
+       return 1;
 }

What are the implications of doing something like this? Is "sync" going to 
stop working or isn't it using this function?

Thanks,

  Francesco

-- 
Dr. Francesco Biscani
Dipartimento di Astronomia
Università di Padova
biscani@pd.astro.it

Re: More Slowdown

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 4113 bytes --]

On Wed, 2005-11-16 at 01:05 -0800, Hans Reiser wrote:
> Craig Shelley wrote:
> 
> >On Tue, 2005-11-15 at 19:04 +0100, Laurent Riffard wrote:
> >  
> >
> >>Please, could you do it again with the -T option for strace? It will
> >>show the time spent in system  calls.
> >>
> >>    
> >>
> >craig@teratron.lan.etheus.net:~$ strace -T -p 8002 2>&1 | grep fsync
> >fsync(26)                               = 0 <0.566159>
> >fsync(26)                               = 0 <0.159852>
> >fsync(26)                               = 0 <0.179659>
> >fsync(26)                               = 0 <0.153406>
> >fsync(26)                               = 0 <0.136685>
> >fsync(26)                               = 0 <0.171057>
> >fsync(26)                               = 0 <0.165733>
> >fsync(26)                               = 0 <0.178846>
> >fsync(26)                               = 0 <0.186595>
> >fsync(26)                               = 0 <0.184368>
> >fsync(26)                               = 0 <0.177273>
> >fsync(26)                               = 0 <0.178634>
> >fsync(26)                               = 0 <0.181036>
> >fsync(26)                               = 0 <0.178750>
> >fsync(26)                               = 0 <0.324480>
> >fsync(26)                               = 0 <0.177226>
> >fsync(26)                               = 0 <0.184625>
> >fsync(26)                               = 0 <0.186224>
> >fsync(26)                               = 0 <0.177282>
> >fsync(26)                               = 0 <0.177146>
> >fsync(26)                               = 0 <0.176000>
> >fsync(26)                               = 0 <0.192434>
> >fsync(26)                               = 0 <0.177482>
> >fsync(26)                               = 0 <0.183357>
> >fsync(26)                               = 0 <0.176528>
> >fsync(26)                               = 0 <0.184334>
> >fsync(26)                               = 0 <0.176622>
> >fsync(26)                               = 0 <0.177977>
> >fsync(26)                               = 0 <0.178636>
> >fsync(26)                               = 0 <0.259863>
> >fsync(26)                               = 0 <0.172789>
> >fsync(26)                               = 0 <0.171739>
> >fsync(26)                               = 0 <0.179641>
> >fsync(26)                               = 0 <0.178157>
> >fsync(26)                               = 0 <0.180272>
> >fsync(26)                               = 0 <0.170426>
> >fsync(26)                               = 0 <0.187630>
> >fsync(26)                               = 0 <0.171589>
> >fsync(26)                               = 0 <0.178135>
> >
> >
> >I seem to be finding that the performance of the system degrades with
> >time and file system usage.
> >Openoffice really starts to run slow after a while. Just as an
> >experiment, I did 
> >ls -R /usr
> >The directory listing appeared very quickly with hardly any hard disk
> >access. Then shortly later when switching back to Openoffice, the
> >process froze for about 3 minutes with continuous hard disk access.
> >
> >One thing which may be relevant is that my kernel is patched with
> >Suspend2 and FBsplash, although I can't remember having any problems
> >like this before.
> >
> >Many Thanks,
> >
> >  
> >
> could you define what the numbers mean (what units, what is measured)?

I believe the value inside < > is the time spent in the system call in
seconds. In the above list, evolution seemed to be calling fsync() as
fast as it could. Although the fd remains at 26 in this case, it closes
and re-opens the file between calls to fsync().
I seem to notice that the length of time spent in the fsync() calls
depends to some degree on how much hard disk activity there has been
since the last call. The strange thing is this seems to apply to reading
also, so performing an ls -R /, and stopping it after a few seconds.
Shortly later, some process will block for tens of seconds when it next
calls fsync().

-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More Slowdown

[Nikita Danilov]

Francesco Biscani writes:
 > On Wednesday 16 November 2005 01:45, David Masover wrote:
 > > I got sick of waiting for it and nuked the fsync call.  All my kernels
 > > have a custom patch such that sys_fsync just returns true, no matter what.
 > 
 > Mhh.. would it be something like this?
 > 
 > --- buffer.c.old        2005-11-16 02:36:46.129829994 +0100
 > +++ buffer.c    2005-11-16 02:37:11.125079752 +0100
 > @@ -376,7 +376,7 @@
 > 
 >  asmlinkage long sys_fsync(unsigned int fd)
 >  {
 > -       return do_fsync(fd, 0);
 > +       return 1;
 >  }
 > 
 > What are the implications of doing something like this? Is "sync" going to 

One implication is following:

 1 application like fetchmail downloads a mail message from the server

 2 saves message in the mailbox

 3 fsyncs the mailbox (which is a no-op in our case)

 4 sends notification to the server, which deletes message

 5 crash occurs (transaction made on the step 2 is not yet committed to
 the disk)

 6 after reboot mailbox is restored to the state it had before step 2

 7 message is lost.

 > stop working or isn't it using this function?
 > 
 > Thanks,
 > 
 >   Francesco

Nikita.

Re: More Slowdown

[John Gilmore]

That "return 1;" should be "return 0;"

a return value of 1 indicates failure.

Also, maybe this should be applied also to the 

asmlinkage long sys_fdatasync(unsigned int fd)

function as well? There isn't much to choose from between these two 
functions...

On Wednesday 16 November 2005 12:18, Nikita Danilov wrote:
> Francesco Biscani writes:
>  > On Wednesday 16 November 2005 01:45, David Masover wrote:
>  > > I got sick of waiting for it and nuked the fsync call.  All my kernels
>  > > have a custom patch such that sys_fsync just returns true, no matter
>  > > what.
>  >
>  > Mhh.. would it be something like this?
>  >
>  > --- buffer.c.old        2005-11-16 02:36:46.129829994 +0100
>  > +++ buffer.c    2005-11-16 02:37:11.125079752 +0100
>  > @@ -376,7 +376,7 @@
>  >
>  >  asmlinkage long sys_fsync(unsigned int fd)
>  >  {
>  > -       return do_fsync(fd, 0);
>  > +       return 1;
>  >  }
>  >
>  > What are the implications of doing something like this? Is "sync" going
>  > to
>
> One implication is following:
>
>  1 application like fetchmail downloads a mail message from the server
>
>  2 saves message in the mailbox
>
>  3 fsyncs the mailbox (which is a no-op in our case)
>
>  4 sends notification to the server, which deletes message
>
>  5 crash occurs (transaction made on the step 2 is not yet committed to
>  the disk)
>
>  6 after reboot mailbox is restored to the state it had before step 2
>
>  7 message is lost.
>
>  > stop working or isn't it using this function?
>  >
>  > Thanks,
>  >
>  >   Francesco
>
> Nikita.

Re: More Slowdown

[David Masover]

Nikita Danilov wrote:
> Francesco Biscani writes:
>  > On Wednesday 16 November 2005 01:45, David Masover wrote:
>  > > I got sick of waiting for it and nuked the fsync call.  All my kernels
>  > > have a custom patch such that sys_fsync just returns true, no matter what.
>  > 
>  > Mhh.. would it be something like this?
>  > 
>  > --- buffer.c.old        2005-11-16 02:36:46.129829994 +0100
>  > +++ buffer.c    2005-11-16 02:37:11.125079752 +0100
>  > @@ -376,7 +376,7 @@
>  > 
>  >  asmlinkage long sys_fsync(unsigned int fd)
>  >  {
>  > -       return do_fsync(fd, 0);
>  > +       return 1;
>  >  }
>  > 
>  > What are the implications of doing something like this? Is "sync" going to 

Yes, sync still works.  That, or the "sync" command has found an
entirely new way of wasting my disk bandwidth.

fsync != sync.

I also sincerely hope that no unmounting, hibernation, or other place
the system _needs_ to sync or data WILL be lost depends on fsync.  Then
again, that shouldn't matter -- the kernel seems to have fsync and
sys_fsync as separate functions.

> One implication is following:
> 
>  1 application like fetchmail downloads a mail message from the server
> 
>  2 saves message in the mailbox
> 
>  3 fsyncs the mailbox (which is a no-op in our case)
> 
>  4 sends notification to the server, which deletes message
> 
>  5 crash occurs (transaction made on the step 2 is not yet committed to
>  the disk)
> 
>  6 after reboot mailbox is restored to the state it had before step 2
> 
>  7 message is lost.

Which is why I refuse to submit my patch.  It's dangerous and shouldn't
be needed, and considering that many users don't even bother to unmount
flash drives in Windows, can you imagine how dangerous it'd be to make
them run "sync" after every time they save an important document?


Some apps do have a legitimate use for fsync, though I wish it was more
configurable -- for instance, vim, mailservers, etc.  Some don't.
Evolution fsyncing every pixel you drag a column is flat-out retarded,
and I'm glad I never needed anything beyond Thunderbird.

Re: More Slowdown

[Artur Makówka]

Artur Makówka wrote:
> My slowdown problems are on my web server, so its concerning different 
> things than Evolution. but this slowdown was very easy to notice when 
> writing something with vim.
> 
> Anyways, i remembered that i was using stable kernel without this 
> happening some time ago. So i tested 2.6.14.x , 2.6.13.x mm and 2.6.13.x 
> , and i still had system crashes. so i downgraded to 2.6.12.6 and the 
> system is stable for more than 24 hours (first time in this month or 
> even longer!)
> 
> probably some processes were doing fsync often (or something similar) 
> and the effect was raising LA and lock-up at the end.
> 
> anyways, 2.6.12.6 (but other than .6 probably too) is, i think, bug 
> free. maybe this will help.

ok, now im 100% sure that 2.6.12.6 kernel + 2.6.12-3 reiser4 patch 
doesnt have this fsync bug, or at least its not causing any big 
slowdowns. The difference is really noticable.

I don't know if i can do anything more to help you fix this bug. But do 
you know is it going to be fixed soon ?

Thanks in advance

Re: More Slowdown

[evilninja]

Francesco Biscani schrieb:
> What are the implications of doing something like this? Is "sync" going to 
> stop working or isn't it using this function?

sync(8) won't stop working, because it's using sync(2):

% strace -T sync 2>&1 | grep sync
execve("/bin/sync", ["sync"], [/* 20 vars */]) = 0
sync()                                  = 0 <0.213305>


...but can someone explain to me: what's the difference between sync(2) 
and fsync(2) ? why are there 2 systemcalls, why is fsync(2) expensive 
and (mis?)used so often by applications and sync(2) is cheap (not?) and 
used only by sync(8)? the manpages say:

fsync, fdatasync -
	synchronize a file's complete in-core state with that on disk

sync -
	commit buffer cache to disk

but both syscalls do not guarantee  data  integrity anyway, because 
"modern disks have large caches".

thank you,
Christian.
-- 
BOFH excuse #453:

Spider infestation in warm case parts

Re: More Slowdown

[michael chang]

On 11/15/05, Craig Shelley <craig@microtron.org.uk> wrote:
> On Mon, 2005-11-14 at 21:47 +0100, Christian Iversen wrote:
> > That's _serisouly_ odd. I've seen something like this happen before, when X
> > programs generated some (non-harmful) X warnings. These were then written
> > into a log file, typically .xsession-errors or similar. Any chance that's
> > what you are seeing?
> >
>
> This explains the problem with Evolution, it looks a bit trigger happy
> with the fsync call while resizing columns.

> Why does calling fsync on this one small file cause so much hard disk
> activity?

If memory serves me right, it has been said time and time again (in
docs and on the list) that fsync performance on Reiser4 is horrendous
because it is (at the moment) unoptimized until ... other issues are
resolved and the optimizations can be implemented.

You should also realized the _reason_ why a majority of disk calls
take so little time - it's because Reiser4 puts them in memory, before
flushing to disk when memory is full.  That would explain any
jerkiness -- it's because Reiser4 is writing everything from the last
real write to the actual disk to now (the current real right) in one
go, and making everything wait.  (Now why this can't be done in
parallel is beyond me - or is it already done in parallel?)

This also means that if your hard disk can't handle writing the
contents of say 80% of your memory's size to disk or it takes 5
seconds for it to do so would explain the problem with the concepts
shown here.  Example: 1GB memory, hard disk can write at a max of 10
MB/s.  If it used 75% of the memory for a Reiser4 cache, [768 MB] it
would take maybe 77 or more seconds to flush. [Yes, a full minute.]  
For comparative purposes, I hear of consumer systems with 2 or 4 GB of
ram, and I know of brand new PC hard disks which have a throughput of
less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
you know the throughputs of your drives, and the size of the memory in
your machines?

When fsyncing, Reiser4, to my understanding, isn't allowed to put
stuff in it's memory cache - it must put it to disk right away.  And
maybe that's also why (partially) Reiser4 performance isn't as optimal
as it will eventually be.  [So, short answer: while there are
shortcomings, and this stuff shouldn't happen, part of this Reiser4FS
behaviour is, to my understanding, intentionally placed.]

Note, this is a my speculation based on what I've read on this mailing
list and on the namesys website, and could be inaccurate.  If any
professionals who are working with the Reiser4 code (e.g. Hans Reiser
and the like) correct me on what I say - their statements take
precedence.

Also, I've noticed in some newer computer's BIOSes (I know some recent
DELL desktop machines have this) sometimes you can find a setting to
change the noisiness/performance trade off of the hard disk.  Maybe
something similar is involved with your PC?  It should be said that
I've witnessed that in quieter mode settings, the disk seems to
operate slower - that could mean that a flush could be less noisy, but
might take longer.

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: More Slowdown

[Hans Reiser]

fsync can be dramatically better optimized, and it will be after the
kernel merge work is completed.  This optimization will likely consist
of reducing the tendency to merge atoms and handling fsync by using
write twice algorithms to a fixed journal area.  Other improvements will
doubtless be found when time is spent on it.  I am sorry that vim and
evolution are suffering so much from our neglecting fsync until after
the merge.  fsync is one of the things I would have us working on if I
had my choice of what we improved in reiser4 rather than other persons
making that choice for us at the moment.  Oh well, such is life in the
society of men.;-)

It looks like akpm is going to start reading the reiser4 code.  I
suspect his remarks will be a step above the ones made so far, he wrote
such nice readahead code.

Re: More Slowdown

[Ingo Bormuth]

If you are frequently hit by fsync (as some of the previosly posters told us) 
an option might be to say something as

  tmgr.atom_max_age=30     (or tmgr.atom_max_size=?)

in fstab (or remount the drive in vulnerable situations).

Of course that sacrifices large parts of reiser4's mean performance, but that
might be better than a largely unusable machine. One day reiser4 will be
fully optimized.




On 2005-11-16 22:08, michael chang wrote:
> That would explain any
> jerkiness -- it's because Reiser4 is writing everything from the last
> real write to the actual disk to now (the current real right) in one
> go, and making everything wait. 


-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517          '(~o-o~)'
public key 86326EC9, http://ibormuth.efil.de/contact     ---ooO--(.)--Ooo---

Re: More Slowdown

[PFC]

	Just a few words about this "slowdown" thing.

	I use linux-2.6.11-cko1-swsusp2 with reiser4 included. I won't upgrade to  
a new version until Hans says the current one is at least as stable as it  
was before starting the merge...

	I get the "slowdown" once in a while, usually for 2-5 seconds, it's not  
that annoying.

	However :

	If I understand well, when memory is needed (memory pressure), reiser4  
triggers a flush and writes dirty pages to the disk in large quantities.
	This has many advantages like not writing at all the temporary files  
which already have been deleted, deferred allocation, etc, it's really  
cool.

	However when "memory pressure" (ie. need to free some RAM) occurs, it is  
usually that I am doing something interactively, like opening a document  
or starting some software. Or it might mean that the database server just  
received a big query which needs some sorting space in RAM for instance.
	Thus, "memory pressure" almost always means "I need memory NOW and  
someone is waiting in front of their screen for it"...
	And it is at this very moment that reiser4 has to flush (to free memory).
	Thus the "flush storm", by nature, always happens when you don't want it  
to happen. It almost never happens when you are away from the computer  
taking a leak, for instance.

	This is analogous to the problem caused to postgres by checkpointing...  
the postgres guys implemented a background writer to solve this.
	I wonder if reiser4 could do the same, ie. trickle down dirty pages to  
free up memory before it is actually needed, to improve reactivity. There  
is a balance to be found, of course, between flushing as late as possible  
(to benefit from all the nifty reiser4 features) and flushing earlier (to  
avoid triggering the "flush storm").
	Maybe this could be set via a few controls...
		- tell reiser4 to try to keep X amount of RAM immediately available  
without flush
		- when the CPU is idle, and/or when the disk is idle, start flushing,  
but stop doing it as soon as some CPU is needed, or a key is hit...
		- just do frequent, small partial flushes which will keep good locality  
of reference while being small
		- do it at a lower priority so that the keyboard does not stop  
responding !!!

	Huh, well, that's it... what do you think ?



> When fsyncing, Reiser4, to my understanding, isn't allowed to put
> stuff in it's memory cache - it must put it to disk right away.  And

	This is the whole point of fsync, yes.
	Now, it's pretty stupid for evlolution to issue a fsync() on every pixel  
move or whatever.
	fsync() is for databases or things which must survive a hard system  
crash...
	evolution could as well have used fflush() and everything would have been  
alright. Dumb.

> For comparative purposes, I hear of consumer systems with 2 or 4 GB of
> ram, and I know of brand new PC hard disks which have a throughput of
> less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
> you know the throughputs of your drives, and the size of the memory in
> your machines?

	Hum. A crap laptop harddrive will do 15 MBytes/s and a recent normal  
desktop drive (7200rpm ATA or SATA) will do 50 MBytes/s or more...
	If you get a lot less (like, 5 MB/s), you have a problem (DMA disabled,  
etc)

Re: More Slowdown

[John Gilmore]

On Thursday 17 November 2005 12:40, Vladimir V. Saveliev wrote:
> Please try whether the attached patch improves anything. It simplifies
> fsync by avoid commiting of transactions which do not modify file being
> fsync-ed.
>
> The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.

I haven't tried this patch yet, but I did try the earlier 'disable fsync 
completely' patch. In fact, I'm using it right now.

It doesn't help. Therefore, your patch probably won't help either.

OK, disabling fsync does help a *little* bit. But I think that the issue (for 
me, anyway) isn't sync per se, it's just flat-out access time. Deleteing lots 
of small files is EXTREMELY slow, but even reading files is slower than it 
should be. It took no less that 10 minutes to delete an old kernel source 
tree, for instance. 

It's related to fragmentation, I think. I didn't really notice any speed 
problems until my hard drive got to about 95% (or so) full. But they haven't 
gone away, even though usage is now down around 54%.

Hrm... I should be able to check that...


About an hour later...

So maybe I was wrong. 6.5% data fragmentation doesn't seem that high. But 
19.8% tree fragmentation does seem a bit high. 

How should this data be interpreted?


#measurefs.reiser4 -T /dev/mapper/e-h -f
measurefs.reiser4 1.0.5
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
reiser4progs/COPYING.

Tree fragmentation ... done
0.197747

#measurefs.reiser4 -S -D /dev/mapper/e-h -f
measurefs.reiser4 1.0.5
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
reiser4progs/COPYING.

Data fragmentation ... done
0.065593
Tree statistics ... done
Packing statistics:
  Formatted nodes:    3721.29b (90.85%)
  Branch nodes:       1734.57b (42.35%)
  Twig nodes:         2886.97b (70.48%)
  Leaf nodes:         3814.82b (93.14%)

Node statistics:
  Total nodes:        8543553
  Formatted nodes:     146354
  Unformatted nodes:  8397199
  Branch nodes:           292
  Twig nodes:           10542
  Leaf nodes:         8532719

Item statistics:
  Total items:         637352
  Nodeptr items:       146353
  Statdata items:     191218
  Direntry items:       17512
  Tail items:          245018
  Extent items:         36869


#

Re: More Slowdown

[Artur Makówka]

Hans Reiser wrote:
> fsync can be dramatically better optimized, and it will be after the
> kernel merge work is completed.  This optimization will likely consist
> of reducing the tendency to merge atoms and handling fsync by using
> write twice algorithms to a fixed journal area.  Other improvements will
> doubtless be found when time is spent on it.  I am sorry that vim and
> evolution are suffering so much from our neglecting fsync until after
> the merge.  fsync is one of the things I would have us working on if I
> had my choice of what we improved in reiser4 rather than other persons
> making that choice for us at the moment.  Oh well, such is life in the
> society of men.;-)
> 
> It looks like akpm is going to start reading the reiser4 code.  I
> suspect his remarks will be a step above the ones made so far, he wrote
> such nice readahead code.
> 

Just want to add, that for desktop systems its only vim and evolution, 
but for my free hosting server it was very frequent system crashes 
(because of that i lost part of my database) and so on. Just want to 
warn anyone deciding to use reiser4 on server, to use 2.6.12 kernels or 
lower as this bug is almost not existent there.

For some uses of reiser4 this bug is very dangerous. Not just slow work 
of some application, but like i said data loss, system instability (few 
crashes every day), and so on.

I would say such a bug/design choice should be top priority, number one. 
Of course thats not true if reiser4 is meant to be only for home use. 
Just want to point that out, because it seems everybody is using reiser4 
only at home, and i am the only one on earth using it on server 
environment (i guess that wasn't the best idea...)

Thanks in advance for consideration of taking this bug to top of your 
TODO list.

Re: More Slowdown

[Vladimir V. Saveliev]

[-- Attachment #1: Type: text/plain, Size: 2145 bytes --]

Hello

Artur Makówka wrote:
> Hans Reiser wrote:
>> fsync can be dramatically better optimized, and it will be after the
>> kernel merge work is completed.  This optimization will likely consist
>> of reducing the tendency to merge atoms and handling fsync by using
>> write twice algorithms to a fixed journal area.  Other improvements will
>> doubtless be found when time is spent on it.  I am sorry that vim and
>> evolution are suffering so much from our neglecting fsync until after
>> the merge.  fsync is one of the things I would have us working on if I
>> had my choice of what we improved in reiser4 rather than other persons
>> making that choice for us at the moment.  Oh well, such is life in the
>> society of men.;-)
>>
>> It looks like akpm is going to start reading the reiser4 code.  I
>> suspect his remarks will be a step above the ones made so far, he wrote
>> such nice readahead code.
>>
> 
> Just want to add, that for desktop systems its only vim and evolution,
> but for my free hosting server it was very frequent system crashes
> (because of that i lost part of my database) and so on. Just want to
> warn anyone deciding to use reiser4 on server, to use 2.6.12 kernels or
> lower as this bug is almost not existent there.
> 
> For some uses of reiser4 this bug is very dangerous. Not just slow work
> of some application, but like i said data loss, system instability (few
> crashes every day), and so on.
> 
> I would say such a bug/design choice should be top priority, number one.
> Of course thats not true if reiser4 is meant to be only for home use.
> Just want to point that out, because it seems everybody is using reiser4
> only at home, and i am the only one on earth using it on server
> environment (i guess that wasn't the best idea...)
> 
> Thanks in advance for consideration of taking this bug to top of your
> TODO list.
> 

Please try whether the attached patch improves anything. It simplifies fsync by
avoid commiting of transactions which do not modify file being fsync-ed.

The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.



[-- Attachment #2: reiser4-fix-fsync.patch --]
[-- Type: text/plain, Size: 3987 bytes --]


This patch simplifies reiser4's fsync. 
It is an experimental patch to test whether it helps against slowdowns reported by users.


 fs/reiser4/plugin/file/file.c |   14 ++++++++++++++
 fs/reiser4/txnmgr.c           |   22 +++++++++++++++-------
 fs/reiser4/txnmgr.h           |    1 +
 3 files changed, 30 insertions(+), 7 deletions(-)

diff -puN fs/reiser4/plugin/file/file.c~reiser4-fix-fsync fs/reiser4/plugin/file/file.c
--- linux-2.6.14-mm2/fs/reiser4/plugin/file/file.c~reiser4-fix-fsync	2005-11-17 13:42:26.000000000 +0300
+++ linux-2.6.14-mm2-vs/fs/reiser4/plugin/file/file.c	2005-11-17 13:42:26.000000000 +0300
@@ -1633,11 +1633,25 @@ int sync_unix_file(struct file *file, st
 {
 	int result;
 	reiser4_context *ctx;
+	txn_atom *atom;
+	reiser4_block_nr reserve;
 
 	ctx = init_context(dentry->d_inode->i_sb);
 	if (IS_ERR(ctx))
 		return PTR_ERR(ctx);
 
+	reserve = estimate_update_common(dentry->d_inode);
+	if (reiser4_grab_space(reserve, BA_CAN_COMMIT))
+		return RETERR(-ENOSPC);
+	write_sd_by_inode_common(dentry->d_inode);
+
+	atom = get_current_atom_locked();
+	spin_lock_txnh(ctx->trans);
+	force_commit_atom(ctx->trans);
+	reiser4_exit_context(ctx);
+	return 0;
+
+
 	assert("nikita-3486", ctx->trans->atom == NULL);
 	result = commit_file_atoms(dentry->d_inode);
 	assert("nikita-3484", ergo(result == 0, ctx->trans->atom == NULL));
diff -puN fs/reiser4/txnmgr.c~reiser4-fix-fsync fs/reiser4/txnmgr.c
--- linux-2.6.14-mm2/fs/reiser4/txnmgr.c~reiser4-fix-fsync	2005-11-17 13:42:26.000000000 +0300
+++ linux-2.6.14-mm2-vs/fs/reiser4/txnmgr.c	2005-11-17 13:42:26.000000000 +0300
@@ -1173,9 +1173,14 @@ static int commit_current_atom(long *nr_
 
 /* TXN_TXNH */
 
-/* commit current atom and wait commit completion; atom and txn_handle should be
- * locked before call, this function unlocks them on exit. */
-static int force_commit_atom_nolock(txn_handle * txnh)
+/**
+ * force_commit_atom - commit current atom and wait commit completion
+ * @txnh:
+ *
+ * Commits current atom and wait commit completion; current atom and @txnh have
+ * to be spinlocked before call, this function unlocks them on exit.
+ */
+int force_commit_atom(txn_handle *txnh)
 {
 	txn_atom *atom;
 
@@ -1188,14 +1193,17 @@ static int force_commit_atom_nolock(txn_
 	assert("zam-834", atom != NULL);
 	assert_spin_locked(&(atom->alock));
 
-	/* Set flags for atom and txnh: forcing atom commit and waiting for
-	 * commit completion */
+	/*
+	 * Set flags for atom and txnh: forcing atom commit and waiting for
+	 * commit completion
+	 */
 	txnh->flags |= TXNH_WAIT_COMMIT;
 	atom->flags |= ATOM_FORCE_COMMIT;
 
 	spin_unlock_txnh(txnh);
 	spin_unlock_atom(atom);
 
+	/* commit is here */
 	txn_restart_current();
 	return 0;
 }
@@ -1240,7 +1248,7 @@ int txnmgr_force_commit_all(struct super
 					spin_lock_txnh(txnh);
 					/* Add force-context txnh */
 					capture_assign_txnh_nolock(atom, txnh);
-					ret = force_commit_atom_nolock(txnh);
+					ret = force_commit_atom(txnh);
 					if (ret)
 						return ret;
 				} else
@@ -2541,7 +2549,7 @@ int sync_atom(txn_atom * atom)
 		if (atom->stage < ASTAGE_PRE_COMMIT) {
 			spin_lock_txnh(txnh);
 			capture_assign_txnh_nolock(atom, txnh);
-			result = force_commit_atom_nolock(txnh);
+			result = force_commit_atom(txnh);
 		} else if (atom->stage < ASTAGE_POST_COMMIT) {
 			/* wait atom commit */
 			atom_wait_event(atom);
diff -puN fs/reiser4/txnmgr.h~reiser4-fix-fsync fs/reiser4/txnmgr.h
--- linux-2.6.14-mm2/fs/reiser4/txnmgr.h~reiser4-fix-fsync	2005-11-17 13:42:26.000000000 +0300
+++ linux-2.6.14-mm2-vs/fs/reiser4/txnmgr.h	2005-11-17 13:42:26.000000000 +0300
@@ -416,6 +416,7 @@ extern int current_atom_should_commit(vo
 extern jnode *find_first_dirty_jnode(txn_atom *, int);
 
 extern int commit_some_atoms(txn_mgr *);
+extern int force_commit_atom(txn_handle *);
 extern int flush_current_atom(int, long, long *, txn_atom **, jnode *);
 
 extern int flush_some_atom(jnode *, long *, const struct writeback_control *, int);

_

Re: More Slowdown

[Hesse, Christian]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

Vladimir V. Saveliev wrote:
> Please try whether the attached patch improves anything. It simplifies
> fsync by avoid commiting of transactions which do not modify file being
> fsync-ed. 
>
> The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.

Hi everybody,

I'm suffering the same problem, sync and fsync are horribly slow. I've written 
a small test program:

http://www.earthworm.de/tmp/reiser4-fsync.c

with 2.6.13:
sync()                                  = 0 <0.000198>
fsync(3)                                = 0 <0.003353>

with 2.6.14 (with and without patch):
sync()                                  = 0 <2.092873>
fsync(3)                                = 0 <0.132579>
-- 
Christian

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: More Slowdown

[Hans Reiser]

PFC wrote:

>
>     Just a few words about this "slowdown" thing.
>
>     I use linux-2.6.11-cko1-swsusp2 with reiser4 included. I won't
> upgrade to  a new version until Hans says the current one is at least
> as stable as it  was before starting the merge...
>
>     I get the "slowdown" once in a while, usually for 2-5 seconds,
> it's not  that annoying.
>
>     However :
>
>     If I understand well, when memory is needed (memory pressure),
> reiser4  triggers a flush and writes dirty pages to the disk in large
> quantities.
>     This has many advantages like not writing at all the temporary
> files  which already have been deleted, deferred allocation, etc, it's
> really  cool.
>
>     However when "memory pressure" (ie. need to free some RAM) occurs,
> it is  usually that I am doing something interactively, like opening a
> document  or starting some software. Or it might mean that the
> database server just  received a big query which needs some sorting
> space in RAM for instance.
>     Thus, "memory pressure" almost always means "I need memory NOW
> and  someone is waiting in front of their screen for it"...
>     And it is at this very moment that reiser4 has to flush (to free
> memory).
>     Thus the "flush storm", by nature, always happens when you don't
> want it  to happen. It almost never happens when you are away from the
> computer  taking a leak, for instance.
>
>     This is analogous to the problem caused to postgres by
> checkpointing...  the postgres guys implemented a background writer to
> solve this.
>     I wonder if reiser4 could do the same, ie. trickle down dirty
> pages to  free up memory before it is actually needed, to improve
> reactivity. There  is a balance to be found, of course, between
> flushing as late as possible  (to benefit from all the nifty reiser4
> features) and flushing earlier (to  avoid triggering the "flush storm").
>     Maybe this could be set via a few controls...
>         - tell reiser4 to try to keep X amount of RAM immediately
> available  without flush
>         - when the CPU is idle, and/or when the disk is idle, start
> flushing,  but stop doing it as soon as some CPU is needed, or a key
> is hit...
>         - just do frequent, small partial flushes which will keep good
> locality  of reference while being small
>         - do it at a lower priority so that the keyboard does not
> stop  responding !!!

I agree that smoothness needs working on.  After we merge.

>
>     Huh, well, that's it... what do you think ?
>
>
>
>> When fsyncing, Reiser4, to my understanding, isn't allowed to put
>> stuff in it's memory cache - it must put it to disk right away.  And
>
>
>     This is the whole point of fsync, yes.
>     Now, it's pretty stupid for evlolution to issue a fsync() on every
> pixel  move or whatever.
>     fsync() is for databases or things which must survive a hard
> system  crash...
>     evolution could as well have used fflush() and everything would
> have been  alright. Dumb.
>
>> For comparative purposes, I hear of consumer systems with 2 or 4 GB of
>> ram, and I know of brand new PC hard disks which have a throughput of
>> less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
>> you know the throughputs of your drives, and the size of the memory in
>> your machines?
>
>
>     Hum. A crap laptop harddrive will do 15 MBytes/s and a recent
> normal  desktop drive (7200rpm ATA or SATA) will do 50 MBytes/s or
> more...
>     If you get a lot less (like, 5 MB/s), you have a problem (DMA
> disabled,  etc)
>
>
>
>
PFC is right.

Re: More Slowdown

[Hans Reiser]

6% fragmentation is enormous.  You very much need the repacker we have
not yet written.....

remember that one seek and rotate takes ~12 ms, and during that time you
could transfer 50MB*.012 bytes.....

Hans

John Gilmore wrote:

>On Thursday 17 November 2005 12:40, Vladimir V. Saveliev wrote:
>  
>
>>Please try whether the attached patch improves anything. It simplifies
>>fsync by avoid commiting of transactions which do not modify file being
>>fsync-ed.
>>
>>The patch applied to 2.6.14-mm2 with warnings, but that can be ignored.
>>    
>>
>
>I haven't tried this patch yet, but I did try the earlier 'disable fsync 
>completely' patch. In fact, I'm using it right now.
>
>It doesn't help. Therefore, your patch probably won't help either.
>
>OK, disabling fsync does help a *little* bit. But I think that the issue (for 
>me, anyway) isn't sync per se, it's just flat-out access time. Deleteing lots 
>of small files is EXTREMELY slow, but even reading files is slower than it 
>should be. It took no less that 10 minutes to delete an old kernel source 
>tree, for instance. 
>
>It's related to fragmentation, I think. I didn't really notice any speed 
>problems until my hard drive got to about 95% (or so) full. But they haven't 
>gone away, even though usage is now down around 54%.
>
>Hrm... I should be able to check that...
>
>
>About an hour later...
>
>So maybe I was wrong. 6.5% data fragmentation doesn't seem that high. But 
>19.8% tree fragmentation does seem a bit high. 
>
>How should this data be interpreted?
>
>
>#measurefs.reiser4 -T /dev/mapper/e-h -f
>measurefs.reiser4 1.0.5
>Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
>reiser4progs/COPYING.
>
>Tree fragmentation ... done
>0.197747
>
>#measurefs.reiser4 -S -D /dev/mapper/e-h -f
>measurefs.reiser4 1.0.5
>Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by 
>reiser4progs/COPYING.
>
>Data fragmentation ... done
>0.065593
>Tree statistics ... done
>Packing statistics:
>  Formatted nodes:    3721.29b (90.85%)
>  Branch nodes:       1734.57b (42.35%)
>  Twig nodes:         2886.97b (70.48%)
>  Leaf nodes:         3814.82b (93.14%)
>
>Node statistics:
>  Total nodes:        8543553
>  Formatted nodes:     146354
>  Unformatted nodes:  8397199
>  Branch nodes:           292
>  Twig nodes:           10542
>  Leaf nodes:         8532719
>
>Item statistics:
>  Total items:         637352
>  Nodeptr items:       146353
>  Statdata items:     191218
>  Direntry items:       17512
>  Tail items:          245018
>  Extent items:         36869
>
>
>#
>
>
>  
>

Re: More Slowdown

[Hans Reiser]

I would like to thank the users for doing so much to help us isolate
this bug.

Vladimir, it seems pretty clear that the problem is due to a code
change, not due to a lack of a feature.  Please isolate it to the
responsible code change, and fix the code change, before adding any
other code changes to speed things up.

Re: More Slowdown

[michael chang]

On 11/17/05, Hans Reiser <reiser@namesys.com> wrote:
> PFC wrote:
>
> >
> >     Just a few words about this "slowdown" thing.
> >
> >     I use linux-2.6.11-cko1-swsusp2 with reiser4 included. I won't
> > upgrade to  a new version until Hans says the current one is at least
> > as stable as it  was before starting the merge...
> >
> >     I get the "slowdown" once in a while, usually for 2-5 seconds,
> > it's not  that annoying.
> >
> >     However :
> >
> >     If I understand well, when memory is needed (memory pressure),
> > reiser4  triggers a flush and writes dirty pages to the disk in large
> > quantities.
> >     This has many advantages like not writing at all the temporary
> > files  which already have been deleted, deferred allocation, etc, it's
> > really  cool.
> >
> >     However when "memory pressure" (ie. need to free some RAM) occurs,
> > it is  usually that I am doing something interactively, like opening a
> > document  or starting some software. Or it might mean that the
> > database server just  received a big query which needs some sorting
> > space in RAM for instance.
> >     Thus, "memory pressure" almost always means "I need memory NOW
> > and  someone is waiting in front of their screen for it"...
> >     And it is at this very moment that reiser4 has to flush (to free
> > memory).
> >     Thus the "flush storm", by nature, always happens when you don't
> > want it  to happen. It almost never happens when you are away from the
> > computer  taking a leak, for instance.
> >
> >     This is analogous to the problem caused to postgres by
> > checkpointing...  the postgres guys implemented a background writer to
> > solve this.
> >     I wonder if reiser4 could do the same, ie. trickle down dirty
> > pages to  free up memory before it is actually needed, to improve
> > reactivity. There  is a balance to be found, of course, between
> > flushing as late as possible  (to benefit from all the nifty reiser4
> > features) and flushing earlier (to  avoid triggering the "flush storm").
> >     Maybe this could be set via a few controls...
> >         - tell reiser4 to try to keep X amount of RAM immediately
> > available  without flush
> >         - when the CPU is idle, and/or when the disk is idle, start
> > flushing,  but stop doing it as soon as some CPU is needed, or a key
> > is hit...
> >         - just do frequent, small partial flushes which will keep good
> > locality  of reference while being small
> >         - do it at a lower priority so that the keyboard does not
> > stop  responding !!!
>
> I agree that smoothness needs working on.  After we merge.
>
> >
> >     Huh, well, that's it... what do you think ?
> >
> >
> >
> >> When fsyncing, Reiser4, to my understanding, isn't allowed to put
> >> stuff in it's memory cache - it must put it to disk right away.  And
> >
> >
> >     This is the whole point of fsync, yes.
> >     Now, it's pretty stupid for evlolution to issue a fsync() on every
> > pixel  move or whatever.
> >     fsync() is for databases or things which must survive a hard
> > system  crash...
> >     evolution could as well have used fflush() and everything would
> > have been  alright. Dumb.
> >
> >> For comparative purposes, I hear of consumer systems with 2 or 4 GB of
> >> ram, and I know of brand new PC hard disks which have a throughput of
> >> less than 5 MB/s (my hard disk maxees out at 20 MB/s on my PC.)  Do
> >> you know the throughputs of your drives, and the size of the memory in
> >> your machines?
> >
> >
> >     Hum. A crap laptop harddrive will do 15 MBytes/s and a recent
> > normal  desktop drive (7200rpm ATA or SATA) will do 50 MBytes/s or
> > more...
> >     If you get a lot less (like, 5 MB/s), you have a problem (DMA
> > disabled,  etc)
> >
> >
> >
> >
> PFC is right.
>

*shrugs* 15-30 MB/s on a recent low-end desktop from Dell (not that
old, maybe 6-12 months) that I own with DMA enabled; the other
machines are because people I know are using older PCs and can't
afford to upgrade them at the moment.  (Ironically, sometimes, those
PCs /feel/ faster, for some reason...)  Of course, a 7200rpm ATA/SATA
drive would be the best; but I just try and make do, as do people I
know.  But that's life; there will always be those on the bottom of
the heap. :)

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: More Slowdown

[Marcel Hilzinger]

Am Donnerstag, 17. November 2005 14:47 schrieb Hesse, Christian:
> Vladimir V. Saveliev wrote:
> > Please try whether the attached patch improves anything. It simplifies
> > fsync by avoid commiting of transactions which do not modify file being
> > fsync-ed.

Did you ever think about, that this is not a reiser4 problem, but a kernel bug 
or a problem related to hal? 

Suse 10 has big problems with external USB drives using sync as mount option. 
They used sync already in 9.3 but with 2.6.11 there were no such problems. 
There has been some changes in kernel 2.6.13 and 2.6.14 concering the sync 
behavior, perhaps it can help investigating there first, bevor fixing reiser4 
for nothing...

-- 
Üdvözlettel -- Mit freundlichen Grüssen,
Marcel Hilzinger

Re: More Slowdown

[Sander]

Marcel Hilzinger wrote (ao):
> Did you ever think about, that this is not a reiser4 problem, but a
> kernel bug or a problem related to hal? 
> 
> Suse 10 has big problems with external USB drives using sync as mount
> option. They used sync already in 9.3 but with 2.6.11 there were no
> such problems. There has been some changes in kernel 2.6.13 and 2.6.14
> concering the sync behavior, perhaps it can help investigating there
> first, bevor fixing reiser4 for nothing...

Running Debian here, and all the reports have been on Reiser4
filesystems. Also, the same system has no such problems with other
filesystems.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: More Slowdown

[Marcel Hilzinger]

Am Dienstag, 22. November 2005 11:38 schrieb Sander:
> Marcel Hilzinger wrote (ao):
> > Did you ever think about, that this is not a reiser4 problem, but a
> > kernel bug or a problem related to hal?
> >
> > Suse 10 has big problems with external USB drives using sync as mount
> > option. They used sync already in 9.3 but with 2.6.11 there were no
> > such problems. There has been some changes in kernel 2.6.13 and 2.6.14
> > concering the sync behavior, perhaps it can help investigating there
> > first, bevor fixing reiser4 for nothing...
>
> Running Debian here, and all the reports have been on Reiser4
> filesystems. Also, the same system has no such problems with other
> filesystems.
It's not about Debian or Suse. It's about kernel 2.6.13 or 2.6.14. At least it 
seems, that the problem does not appear on older kernels (right?)

-- 
Üdvözlettel -- Mit freundlichen Grüssen,
Marcel Hilzinger

Re: More Slowdown

[Ingo Bormuth]

On 2005-11-22 11:23, Marcel Hilzinger wrote:
> 
> Did you ever think about, that this is not a reiser4 problem, but a kernel bug 
> or a problem related to hal? 
> 

Just to mention again:

I do not see the problem on my Laptop. I use a the clean vanilla-2.6.14.2 
kernel manually patched with reiser-for-2.6.14-1. It's a slow laptop with
broken DMA so I should realize if there was something wrong.

Afaik every poster complaining so far had mm-kernels, larger patchsets or
additional stuff as swsusp2. So: Do jou see the slowdown/crashes in vanilla ?


-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517
public key 86326EC9, http://ibormuth.efil.de/contact

Re: More Slowdown

[Artur Makówka]

Ingo Bormuth wrote:
> On 2005-11-22 11:23, Marcel Hilzinger wrote:
>> Did you ever think about, that this is not a reiser4 problem, but a kernel bug 
>> or a problem related to hal? 
>>
> 
> Just to mention again:
> 
> I do not see the problem on my Laptop. I use a the clean vanilla-2.6.14.2 
> kernel manually patched with reiser-for-2.6.14-1. It's a slow laptop with
> broken DMA so I should realize if there was something wrong.
> 
> Afaik every poster complaining so far had mm-kernels, larger patchsets or
> additional stuff as swsusp2. So: Do jou see the slowdown/crashes in vanilla ?
> 
> 

yes, like i said several times (and as others also said) this bug is on 
vanilla 2.6.14.2 or 2.6.13.x. The fact that this bug is not on every 
computer doesn't mean it don't exists.

I have big slowdowns on 2.6.13 or 2.6.14 sometimes causing crash - and 
one time it caused database loss. (the bug itself doesnt cause crash i 
suppose,but the traffic i get on this server + this bug can cause crash)

its stable at 2.6.12.6

Re: More Slowdown

[David Masover]

Marcel Hilzinger wrote:
> Am Dienstag, 22. November 2005 11:38 schrieb Sander:
> 
>>Marcel Hilzinger wrote (ao):
>>
>>>Did you ever think about, that this is not a reiser4 problem, but a
>>>kernel bug or a problem related to hal?
>>>
>>>Suse 10 has big problems with external USB drives using sync as mount
>>>option. They used sync already in 9.3 but with 2.6.11 there were no
>>>such problems. There has been some changes in kernel 2.6.13 and 2.6.14
>>>concering the sync behavior, perhaps it can help investigating there
>>>first, bevor fixing reiser4 for nothing...
>>
>>Running Debian here, and all the reports have been on Reiser4
>>filesystems. Also, the same system has no such problems with other
>>filesystems.
> 
> It's not about Debian or Suse. It's about kernel 2.6.13 or 2.6.14. At least it 
> seems, that the problem does not appear on older kernels (right?)

Wrong.

You are talking about the fsync issue, right?  As far as I know, while
there has been progress lately, fsync has always been slow on Reiser4,
because until recently, it was basically a call to sync, and reiser4
syncs can be huge due to lazy writes -- stuff only ever hits disk when
there's nowhere else to put it in RAM.  Actual calls to sync are rare
enough (shutdown/reboot) that lazy writes are a good thing, but fsync
apparently needs to be faster.

I disabled fsync before the FS was even stable, because I was sick of
waiting 30 seconds or so for vim to save and quit.

It may help to have fsync only sync the file in question (as it always
has, except on Reiser4).  This has been done with lazy writes, in XFS,
so I see no reason it can't be done here -- there might have even been a
patch recently.  Personally, I'd like to see it stay as slow as it is
for awhile, so we can find the people doing stupid things (evolution)
and flame them into crispy obediance.

fsync means flush to disk.  This is something you're supposed to do with
a file when the file is so important you want to guarentee you'll have
the most recent, uncorrupted version after a crash.  If evolution is
calling fsync while resizing, this is an evolution bug, made more
obvious by reiser4 -- if a computer crashes, does the user really care
if their Evolution columns are still lined up _exactly_ the way they
were (maybe mid-click'n'drag) when the crash occurred?

If there is a user who cares that much about column widths, can we flame
them to crispiness also?

Re: More Slowdown

[Marcel Hilzinger]

Am Dienstag, 22. November 2005 15:29 schrieb David Masover:
> Marcel Hilzinger wrote:
[...]
> > It's not about Debian or Suse. It's about kernel 2.6.13 or 2.6.14. At
> > least it seems, that the problem does not appear on older kernels
> > (right?)
>
> Wrong.
>
> You are talking about the fsync issue, right?  As far as I know, while
> there has been progress lately, fsync has always been slow on Reiser4,
> because until recently, it was basically a call to sync, and reiser4
> syncs can be huge due to lazy writes -- stuff only ever hits disk when
> there's nowhere else to put it in RAM.  Actual calls to sync are rare
> enough (shutdown/reboot) that lazy writes are a good thing, but fsync
> apparently needs to be faster.
>
> I disabled fsync before the FS was even stable, because I was sick of
> waiting 30 seconds or so for vim to save and quit.
>
> It may help to have fsync only sync the file in question (as it always
> has, except on Reiser4).  This has been done with lazy writes, in XFS,
> so I see no reason it can't be done here -- there might have even been a
> patch recently.  Personally, I'd like to see it stay as slow as it is
> for awhile, so we can find the people doing stupid things (evolution)
> and flame them into crispy obediance.
>
> fsync means flush to disk.  This is something you're supposed to do with
> a file when the file is so important you want to guarentee you'll have
> the most recent, uncorrupted version after a crash.  If evolution is
> calling fsync while resizing, this is an evolution bug, made more
> obvious by reiser4 -- if a computer crashes, does the user really care
> if their Evolution columns are still lined up _exactly_ the way they
> were (maybe mid-click'n'drag) when the crash occurred?
Thanks a lot. I understand now. But is there any bug to hunt within reiser4 
then?

-- 
Üdvözlettel -- Mit freundlichen Grüssen,
Marcel Hilzinger

Re: More Slowdown

[Sander]

Marcel Hilzinger wrote (ao):
> Am Dienstag, 22. November 2005 11:38 schrieb Sander:
> > Marcel Hilzinger wrote (ao):
> > > Did you ever think about, that this is not a reiser4 problem, but a
> > > kernel bug or a problem related to hal?
> > >
> > > Suse 10 has big problems with external USB drives using sync as mount
> > > option. They used sync already in 9.3 but with 2.6.11 there were no
> > > such problems. There has been some changes in kernel 2.6.13 and 2.6.14
> > > concering the sync behavior, perhaps it can help investigating there
> > > first, bevor fixing reiser4 for nothing...
> >
> > Running Debian here, and all the reports have been on Reiser4
> > filesystems. Also, the same system has no such problems with other
> > filesystems.
>
> It's not about Debian or Suse. It's about kernel 2.6.13 or 2.6.14. At least it 
> seems, that the problem does not appear on older kernels (right?)

That is correct, but in your previous mail you suggested this might not
be a Reiser4 problem, and that is incorrect :-)

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: More Slowdown

[David Masover]

Marcel Hilzinger wrote:
> Am Dienstag, 22. November 2005 15:29 schrieb David Masover:
> 
>>Marcel Hilzinger wrote:
> 
> [...]
> 
>>>It's not about Debian or Suse. It's about kernel 2.6.13 or 2.6.14. At
>>>least it seems, that the problem does not appear on older kernels
>>>(right?)
>>
>>Wrong.
>>
>>You are talking about the fsync issue, right?  As far as I know, while
>>there has been progress lately, fsync has always been slow on Reiser4,
>>because until recently, it was basically a call to sync, and reiser4


> Thanks a lot. I understand now. But is there any bug to hunt within reiser4 
> then?

You could call it that.

Last I checked, Reiser4's fsync just called 'sync'.  The way it should
work is to only flush the file being fsynced, not the whole FS.

If this were fixed, Reiser4 fsync performance should approach that of,
say, XFS, which also has lazy writes.  But, I haven't been keeping up
with this discussion, so maybe someone did fix that, and it's still slow.

But, even if fsync is as fast as it can possibly be, Evolution should
also be fixed.

Re: More Slowdown

[Hans Reiser]

Evolution may need tweaking also, but reiser4's fsync clearly needs work.