[PATCH] new data logging and quota patches available

[PATCH] new data logging and quota patches available

[Chris Mason]

Hello all,

ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be
updated with a new set of data logging and quota patches against
2.4.21-pre4

The data logging code is updated with another set of io stalling fixes,
they should improve performance of data=ordered and data=writeback by
being smarter about forcing commits under heavy write load and kicking
kreiserfsd.

Treat these with care, they've gotten a ton of testing under the suse
kernel, but the port to vanilla was just done today.

The quota patches include a fix for incorrect sd_block counts on
symlinks.

-chris

Re: [PATCH] new data logging and quota patches available

[Philippe Gramoullé]

Hi Chris,

Well, this is excellent :)

Does that include the jounal overflow fix ?

Thanks,

Philippe

On 21 Feb 2003 18:32:11 -0500
Chris Mason <mason@suse.com> wrote:


  |  Hello all,
  |  
  |  ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be
  |  updated with a new set of data logging and quota patches against
  |  2.4.21-pre4
  |  
  |  The data logging code is updated with another set of io stalling fixes,
  |  they should improve performance of data=ordered and data=writeback by
  |  being smarter about forcing commits under heavy write load and kicking
  |  kreiserfsd.
  |  
  |  Treat these with care, they've gotten a ton of testing under the suse
  |  kernel, but the port to vanilla was just done today.
  |  
  |  The quota patches include a fix for incorrect sd_block counts on
  |  symlinks.
  |  
  |  -chris
  |  
  |  

Re: [PATCH] new data logging and quota patches available

[Dieter Nützel]

Am Samstag, 22. Februar 2003 00:32 schrieb Chris Mason:
> Hello all,
>
> ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be
> updated with a new set of data logging and quota patches against
> 2.4.21-pre4
>
> The data logging code is updated with another set of io stalling fixes,
> they should improve performance of data=ordered and data=writeback by
> being smarter about forcing commits under heavy write load and kicking
> kreiserfsd.
>
> Treat these with care, they've gotten a ton of testing under the suse
> kernel, but the port to vanilla was just done today.

Which files are needed for 2.4.21-pre4aa3?

Have a nice weekend.

-Dieter

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sat, 2003-02-22 at 07:41, Dieter Nützel wrote:

> Which files are needed for 2.4.21-pre4aa3?

ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21-aa

I just uploaded a fresh set of data logging and quota patches for -aa

-chris

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

And here's one more patch for those of you that want to experiment a
little.  reiserfs_dirty_inode gets called to log the inode every time
the vfs layer does an atime/mtime/ctime update, which is one of hte
reasons mounting with -o noatime and -o nodiratime makes things faster. 
We had to do this because otherwise kswapd can end up trying to write
inodes, which can lead to deadlock as he tries to wait on the log.

One of the patches in my data logging directory is kinoded-8, which
changes things so a new kinoded does all inode writeback instead of
kswapd.

That means that if you apply up to 05-data-logging-36 and then apply
09-kinoded-8 (you won't need any of the other quota patches), you can
also apply this patch.  It changes reiserfs to leave inodes dirty, which
saves us lots of time on atime/mtime updates.

I'l upload this after it gets a little additional testing, but wanted to
include it here in case anyone else was interested in benchmarking.

--- linux.suse.1/fs/reiserfs/inode.c	2003-02-20 11:22:08.000000000 -0500
+++ linux.suse/fs/reiserfs/inode.c	2003-02-20 16:13:28.000000000 -0500
@@ -1492,12 +1498,19 @@
 ** does something when called for a synchronous update.
 */
 void reiserfs_write_inode (struct inode * inode, int do_sync) {
-
+    struct reiserfs_transaction_handle th;
     if (inode->i_sb->s_flags & MS_RDONLY) {
         reiserfs_warning("clm-6005: writing inode %lu on readonly FS\n", 
 	                  inode->i_ino) ;
         return ;
     }
+
+    lock_kernel() ;
+    journal_begin(&th, inode->i_sb, 1) ;
+    reiserfs_update_sd (&th, inode);
+    journal_end(&th, inode->i_sb, 1) ;
+    unlock_kernel() ;
+
     /* memory pressure can sometimes initiate write_inode calls with sync == 1,
     ** these cases are just when the system needs ram, not when the 
     ** inode needs to reach disk for safety, and they can safely be
--- linux.suse.1/fs/reiserfs/super.c	2003-02-20 11:22:08.000000000 -0500
+++ linux.suse/fs/reiserfs/super.c	2003-02-20 20:07:06.000000000 -0500
@@ -421,15 +422,18 @@
 	                  inode->i_ino) ;
         return ;
     }
-    lock_kernel() ;
-
-    /* this is really only used for atime updates, so they don't have
-    ** to be included in O_SYNC or fsync
+    /* this is used for atime updates, so they don't have
+    ** to be included in O_SYNC or fsync, but if we are nesting in to
+    ** a transaction, journal_begin/end is free, so we might as well
+    ** log the inode now (plus quota depends on it)
     */
-    journal_begin(&th, inode->i_sb, 1) ;
-    reiserfs_update_sd (&th, inode);
-    journal_end(&th, inode->i_sb, 1) ;
-    unlock_kernel() ;
+    if (reiserfs_transaction_running(inode->i_sb)) {
+	lock_kernel() ;
+	journal_begin(&th, inode->i_sb, 1) ;
+	reiserfs_update_sd (&th, inode);
+	journal_end(&th, inode->i_sb, 1) ;
+	unlock_kernel() ;
+    }
 }
 
 static void reiserfs_clear_inode (struct inode *inode)


 

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/22/2003 05:12 PM, Chris Mason wrote:
> And here's one more patch for those of you that want to experiment a
> little.  reiserfs_dirty_inode gets called to log the inode every time
> the vfs layer does an atime/mtime/ctime update, which is one of hte
> reasons mounting with -o noatime and -o nodiratime makes things faster. 
> We had to do this because otherwise kswapd can end up trying to write
> inodes, which can lead to deadlock as he tries to wait on the log.
> 
> One of the patches in my data logging directory is kinoded-8, which
> changes things so a new kinoded does all inode writeback instead of
> kswapd.
> 
> That means that if you apply up to 05-data-logging-36 and then apply
> 09-kinoded-8 (you won't need any of the other quota patches), you can
> also apply this patch.  It changes reiserfs to leave inodes dirty, which
> saves us lots of time on atime/mtime updates.
> 
> I'l upload this after it gets a little additional testing, but wanted to
> include it here in case anyone else was interested in benchmarking.
> 
> [11.dirty-inodes-for-kinoded.diff]

Hi Chris!

At least I'm not able to copy through my partition via "cp -a ..." with 
the last supposed setup + preempt without quota ...

When umounting the destination partition it says "device is busy".

I don't know what is gone for now on my repository partition, but I know 
from earlier times this should not happen.

Bye,

Manuel

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/23/2003 01:50 AM, Manuel Krause wrote:
> On 02/22/2003 05:12 PM, Chris Mason wrote:
> 
>> And here's one more patch for those of you that want to experiment a
>> little.  reiserfs_dirty_inode gets called to log the inode every time
>> the vfs layer does an atime/mtime/ctime update, which is one of hte
>> reasons mounting with -o noatime and -o nodiratime makes things 
>> faster. We had to do this because otherwise kswapd can end up trying 
>> to write
>> inodes, which can lead to deadlock as he tries to wait on the log.
>>
>> One of the patches in my data logging directory is kinoded-8, which
>> changes things so a new kinoded does all inode writeback instead of
>> kswapd.
>>
>> That means that if you apply up to 05-data-logging-36 and then apply
>> 09-kinoded-8 (you won't need any of the other quota patches), you can
>> also apply this patch.  It changes reiserfs to leave inodes dirty, which
>> saves us lots of time on atime/mtime updates.
>>
>> I'l upload this after it gets a little additional testing, but wanted to
>> include it here in case anyone else was interested in benchmarking.
>>
>> [11.dirty-inodes-for-kinoded.diff]
> 
> Hi Chris!
> 
> At least I'm not able to copy through my partition via "cp -a ..." with 
> the last supposed setup + preempt without quota ...
> 
> When umounting the destination partition it says "device is busy".
> 
> I don't know what is gone for now on my repository partition, but I know 
> from earlier times this should not happen.

Sorry, I was incomplete with my report, at least with the following: 
After thinking over the remount strategy for some minutes with a pizza 
the partition was umountable again. I don't have exact values but it was 
under about 15 minutes max. with 3.5GB copied previously.

My setup in fact:
m02-akpm-b_journal_head-1.diff
m03-relocation-6.diff
m04-reiserfs-sync_fs-1.diff
m05-data-logging-36.diff
m06-logging-export.diff
m06-write_times.diff
m09-kinoded-8.diff
m11.inode-dirty-for-kinoded-ML.diff
(prefix m for you)

+ latest preempt patch (what was for 2.4.21-pre1).


Thanks,

Manuel

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sat, 2003-02-22 at 20:04, Manuel Krause wrote:
> On 02/23/2003 01:50 AM, Manuel Krause wrote:
> > On 02/22/2003 05:12 PM, Chris Mason wrote:
> > 
> >> And here's one more patch for those of you that want to experiment a
> >> little.  reiserfs_dirty_inode gets called to log the inode every time
> >> the vfs layer does an atime/mtime/ctime update, which is one of hte
> >> reasons mounting with -o noatime and -o nodiratime makes things 
> >> faster. We had to do this because otherwise kswapd can end up trying 
> >> to write
> >> inodes, which can lead to deadlock as he tries to wait on the log.
> >>
> >> One of the patches in my data logging directory is kinoded-8, which
> >> changes things so a new kinoded does all inode writeback instead of
> >> kswapd.
> >>
> >> That means that if you apply up to 05-data-logging-36 and then apply
> >> 09-kinoded-8 (you won't need any of the other quota patches), you can
> >> also apply this patch.  It changes reiserfs to leave inodes dirty, which
> >> saves us lots of time on atime/mtime updates.
> >>
> >> I'l upload this after it gets a little additional testing, but wanted to
> >> include it here in case anyone else was interested in benchmarking.
> >>
> >> [11.dirty-inodes-for-kinoded.diff]
> > 
> > Hi Chris!
> > 
> > At least I'm not able to copy through my partition via "cp -a ..." with 
> > the last supposed setup + preempt without quota ...
> > 
> > When umounting the destination partition it says "device is busy".
> > 
> > I don't know what is gone for now on my repository partition, but I know 
> > from earlier times this should not happen.
> 
> Sorry, I was incomplete with my report, at least with the following: 
> After thinking over the remount strategy for some minutes with a pizza 
> the partition was umountable again. I don't have exact values but it was 
> under about 15 minutes max. with 3.5GB copied previously.

Hmmm, I think I need to changed around kinoded slightly.  Thanks for
giving it a try...

-chris

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

[-- Attachment #1: Type: text/plain, Size: 110 bytes --]

I've attached a new kinoded-9.diff.gz, it will apply to both -aa and
vanilla.  Please give it a try.

-chris


[-- Attachment #2: kinoded-9.diff.gz --]
[-- Type: application/x-gzip, Size: 1509 bytes --]

Re: [PATCH] new data logging and quota patches available

[Marc-Christian Petersen]

On Sunday 23 February 2003 18:18, Chris Mason wrote:

Hi Chris,

> I've attached a new kinoded-9.diff.gz, it will apply to both -aa and
> vanilla.  Please give it a try.
do you consider telling us the changes? ;)

ciao, Marc

Re: [PATCH] new data logging and quota patches available

[Marc-Christian Petersen]

On Sunday 23 February 2003 18:37, Marc-Christian Petersen wrote:

Hi again,

> > I've attached a new kinoded-9.diff.gz, it will apply to both -aa and
> > vanilla.  Please give it a try.
> do you consider telling us the changes? ;)
_hopefully_ an SMP deadlock while massive disk I/O. :)

ciao, Marc

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sun, 2003-02-23 at 13:39, Marc-Christian Petersen wrote:
> On Sunday 23 February 2003 18:37, Marc-Christian Petersen wrote:
> 
> Hi again,
> 
> > > I've attached a new kinoded-9.diff.gz, it will apply to both -aa and
> > > vanilla.  Please give it a try.
> > do you consider telling us the changes? ;)
> _hopefully_ an SMP deadlock while massive disk I/O. :)

No, the new kinoded-9.diff just changes how kinoded gets woken up, to do
a better job of limiting the number of unused inodes (but still
allocated) on the box.

So, which set of patches were you using when you saw the deadlock?  If
you can reproduce, could you please capture and decode the output of
sysrq-t?

-chris

Re: [PATCH] new data logging and quota patches available

[Marc-Christian Petersen]

On Sunday 23 February 2003 20:33, Chris Mason wrote:

Hi Chris,

> > _hopefully_ an SMP deadlock while massive disk I/O. :)
> No, the new kinoded-9.diff just changes how kinoded gets woken up, to do
> a better job of limiting the number of unused inodes (but still
> allocated) on the box.
> So, which set of patches were you using when you saw the deadlock?  If
> you can reproduce, could you please capture and decode the output of
> sysrq-t?
Unfortunately my SMP box died some time ago (end Dec. 2002) so I cannot 
reproduce this yet. It works great with UP machines but I had bad deadlocks 
with SMP and kinoded-8 which was also in the SuSE 2.4.19 kernel tree.

Let me find some informations I wrote down that time. If I found them I write 
them down here.

ciao, Marc

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/23/2003 06:18 PM, Chris Mason wrote:
> I've attached a new kinoded-9.diff.gz, it will apply to both -aa and
> vanilla.  Please give it a try.
> 
> -chris

Hi and thank you!

I've replaced the old patch. And the problem is gone -- the partitions 
are umountable again after copying.

I see a performance decrease with my copies in data=ordered mode by 
about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% 
decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing 
this? Partly I can blame it on the disk content distribution that 
changed. If I removed the kinoded related patches would the throughput 
rate increase?

Bye,

Manuel

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sun, 2003-02-23 at 20:42, Manuel Krause wrote:
> On 02/23/2003 06:18 PM, Chris Mason wrote:
> > I've attached a new kinoded-9.diff.gz, it will apply to both -aa and
> > vanilla.  Please give it a try.
> > 
> > -chris
> 
> Hi and thank you!
> 
> I've replaced the old patch. And the problem is gone -- the partitions 
> are umountable again after copying.
> 

Good to hear

> I see a performance decrease with my copies in data=ordered mode by 
> about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% 
> decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing 
> this? Partly I can blame it on the disk content distribution that 
> changed. If I removed the kinoded related patches would the throughput 
> rate increase?

That's odd, tests here show performance increasing but I'm testing
2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging

Could I talk you into trying to narrow this down?  Perhaps starting with
a benchmark of the same dataset on both kernels?  The new data logging
code kicks kreiserfsd in sooner to service async commits.  So if your
benchmark lasts < 30 seconds, it is possible the older kernels never did
writes at all during the run.

-chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/24/2003 03:08 PM, Chris Mason wrote:
> On Sun, 2003-02-23 at 20:42, Manuel Krause wrote:
>>On 02/23/2003 06:18 PM, Chris Mason wrote:
>>
[...]
> 
>>I see a performance decrease with my copies in data=ordered mode by 
>>about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% 
>>decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing 
>>this? Partly I can blame it on the disk content distribution that 
>>changed. If I removed the kinoded related patches would the throughput 
>>rate increase?
> 
> That's odd, tests here show performance increasing but I'm testing
> 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
> 
> Could I talk you into trying to narrow this down?  Perhaps starting with
> a benchmark of the same dataset on both kernels?  The new data logging
> code kicks kreiserfsd in sooner to service async commits.  So if your
> benchmark lasts < 30 seconds, it is possible the older kernels never did
> writes at all during the run.

No, the copy usually takes about 9 min. to complete and I umount the 
partitions to take (more) care everything went over correctly.

Yes, I can retest it with the old data-logging, too. But before just one 
clarification for "old data logging": Do you mean just the kinoded 
patches removed or even data-logging-35 instead of -36 ?

Thanks,

Manuel

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/24/2003 06:45 PM, Manuel Krause wrote:
> On 02/24/2003 03:08 PM, Chris Mason wrote:
>> On Sun, 2003-02-23 at 20:42, Manuel Krause wrote:
>>> On 02/23/2003 06:18 PM, Chris Mason wrote:
>>>
> [...]
> 
>>> I see a performance decrease with my copies in data=ordered mode by 
>>> about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% 
>>> decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing 
>>> this? Partly I can blame it on the disk content distribution that 
>>> changed. If I removed the kinoded related patches would the 
>>> throughput rate increase?
>> 
>> That's odd, tests here show performance increasing but I'm testing
>> 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
>> 
>> Could I talk you into trying to narrow this down?  Perhaps starting with
>> a benchmark of the same dataset on both kernels?  The new data logging
>> code kicks kreiserfsd in sooner to service async commits.  So if your
>> benchmark lasts < 30 seconds, it is possible the older kernels never did
>> writes at all during the run.
> 
> No, the copy usually takes about 9 min. to complete and I umount the 
> partitions to take (more) care everything went over correctly.

(And this time is included when I compare the timing values.)
> 
> Yes, I can retest it with the old data-logging, too. But before just one 
> clarification for "old data logging": Do you mean just the kinoded 
> patches removed or even data-logging-35 instead of -36 ?

So you seem to agree...? ;-) O.k.

Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I 
have a performance increase of 2.5% for the new data-logging. Like you 
say above.

Best regards,

Manuel

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/24/2003 11:19 PM, Manuel Krause wrote:
> On 02/24/2003 06:45 PM, Manuel Krause wrote: 
>> On 02/24/2003 03:08 PM, Chris Mason wrote:
>>> On Sun, 2003-02-23 at 20:42, Manuel Krause wrote:
>>>> On 02/23/2003 06:18 PM, Chris Mason wrote: 
>>>>
>> [...]
>>
>>>> I see a performance decrease with my copies in data=ordered mode by 
>>>> about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% 
>>>> decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing 
>>>> this? Partly I can blame it on the disk content distribution that 
>>>> changed. If I removed the kinoded related patches would the 
>>>> throughput rate increase?
>>>
>>> That's odd, tests here show performance increasing but I'm testing
>>> 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
>>>
>>> Could I talk you into trying to narrow this down?  Perhaps starting with
>>> a benchmark of the same dataset on both kernels?  The new data logging
>>> code kicks kreiserfsd in sooner to service async commits.  So if your
>>> benchmark lasts < 30 seconds, it is possible the older kernels never did
>>> writes at all during the run.
>>
>> No, the copy usually takes about 9 min. to complete and I umount the 
>> partitions to take (more) care everything went over correctly.
> 
> (And this time is included when I compare the timing values.)
> 
>> Yes, I can retest it with the old data-logging, too. But before just 
>> one clarification for "old data logging": Do you mean just the kinoded 
>> patches removed or even data-logging-35 instead of -36 ?
> 
> So you seem to agree...? ;-) O.k.
> 
> Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I 
> have a performance increase of 2.5% for the new data-logging. Like you 
> say above.

But... though you were right with 2.4.21 data-logging performance 
relations: --> What was causing that performance loss between 2.4.19 
+data-logging and now???

A usual problem in my mind: Due to performance issues I would like to 
downgrade -- due to recent bugfixes and your additional enhancements, I 
would go with a performance loss of ~16.5%...

Should I really need to retest all kernels without data-logging patches 
on here to get the truth?


Mmmh. No, I think you should know the reasons and explain them when I'm 
just clueless upon my results...


Best regards to you and all others,


Manuel

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

> > Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I 
> > have a performance increase of 2.5% for the new data-logging. Like you 
> > say above.
> 
> But... though you were right with 2.4.21 data-logging performance 
> relations: --> What was causing that performance loss between 2.4.19 
> +data-logging and now???
> 

Did you exclude your changing dataset as a potential cause?  If not I'm
very interested in the details.

-chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 02/28/2003 02:32 PM, Chris Mason wrote:
>>>Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I 
>>>have a performance increase of 2.5% for the new data-logging. Like you 
>>>say above.
>>
>>But... though you were right with 2.4.21 data-logging performance 
>>relations: --> What was causing that performance loss between 2.4.19 
>>+data-logging and now???
>>
> Did you exclude your changing dataset as a potential cause?  If not I'm
> very interested in the details.

Please, tell me how to respond to this question!

I didn't want to _exclude_ it as a potential cause but felt in need to 
write it for the records. The test partition varies from day to day and 
to recreate a fillup rate 71%~73% relating to  my previous tests was 
hard enough. I'm not able to hold a "reiserfs-special-testing-partition" 
on here... Mmh.

Unfortunately I didn't do the file-size-distribution one that time so 
you need to give me further hints on how to test my usual backup copies 
more valuably for you -- but only with these copies.

But whatever I did with that disk content at ~72% usage and staying 
error-free -- YOU can't blame it to the content distribution. Not 19% 
degression in copy rate since 2.4.19. Explain it, please!

... But now we have 2.4.21-pre5 with direct-io and iget5-patches already...


Thank you very much,

Manuel

P.S.: Do the data-logging things go to namesys now? (Me, watching Olegs 
recent engagements...)

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Wed, 2003-03-05 at 21:49, Manuel Krause wrote:
> On 02/28/2003 02:32 PM, Chris Mason wrote:
> >>>Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I 
> >>>have a performance increase of 2.5% for the new data-logging. Like you 
> >>>say above.
> >>
> >>But... though you were right with 2.4.21 data-logging performance 
> >>relations: --> What was causing that performance loss between 2.4.19 
> >>+data-logging and now???
> >>
> > Did you exclude your changing dataset as a potential cause?  If not I'm
> > very interested in the details.
> 
> Please, tell me how to respond to this question!
> 

Sorry, I wasn't clear.  It sounds like you've got two very different
copy speeds on two possibly different data sets.  The problem will be
significantly easier to debug and discuss if you boot a 2.4.19 + data
logging kernel, do the test, and then do the test again on a 2.4.21-pre
data logging kernel.

I know this is a pain, I want to reduce the number of variables in
place.

> P.S.: Do the data-logging things go to namesys now? (Me, watching Olegs 
> recent engagements...)

Oleg is making patches for the -ac series.  He also merged a special set
for Philippe, they pay namesys for support ;-)

I'm remerging my work over the last week and should have a new set of
patches out soon.

-chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 03/06/2003 04:46 AM, Chris Mason wrote:
> On Wed, 2003-03-05 at 21:49, Manuel Krause wrote:
> 
>>On 02/28/2003 02:32 PM, Chris Mason wrote:
>>
>>>>>Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I 
>>>>>have a performance increase of 2.5% for the new data-logging. Like you 
>>>>>say above.
>>>>
>>>>But... though you were right with 2.4.21 data-logging performance 
>>>>relations: --> What was causing that performance loss between 2.4.19 
>>>>+data-logging and now???
>>>>
>>>
>>>Did you exclude your changing dataset as a potential cause?  If not I'm
>>>very interested in the details.
>>
>>Please, tell me how to respond to this question!
>>
> 
> Sorry, I wasn't clear.  It sounds like you've got two very different
> copy speeds on two possibly different data sets.  The problem will be
> significantly easier to debug and discuss if you boot a 2.4.19 + data
> logging kernel, do the test, and then do the test again on a 2.4.21-pre
> data logging kernel.
> 
> I know this is a pain, I want to reduce the number of variables in
> place.
> 

Yes. That all is why I asked. I'll try to find and retest the old things 
tomorrow night. The new setup will go fine with 2.4.21-pre5 ? Or is this 
one more step of irritation? Mmm. Or should I wait for the patches you 
announce below?

Thank you, good "night",

Manuel

> 
>>P.S.: Do the data-logging things go to namesys now? (Me, watching Olegs 
>>recent engagements...)
> 
> Oleg is making patches for the -ac series.  He also merged a special set
> for Philippe, they pay namesys for support ;-)

My additional thanks go to Philippe ! :-)))

> I'm remerging my work over the last week and should have a new set of
> patches out soon.
> 
> -chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 03/06/2003 05:03 AM, Manuel Krause wrote:
> On 03/06/2003 04:46 AM, Chris Mason wrote:
> 
>> On Wed, 2003-03-05 at 21:49, Manuel Krause wrote:
>>
>>> On 02/28/2003 02:32 PM, Chris Mason wrote:
>>>
>>>>>> Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode 
>>>>>> I have a performance increase of 2.5% for the new data-logging. 
>>>>>> Like you say above.
>>>>>
>>>>>
>>>>> But... though you were right with 2.4.21 data-logging performance 
>>>>> relations: --> What was causing that performance loss between 
>>>>> 2.4.19 +data-logging and now???
>>>>>
>>>>
>>>> Did you exclude your changing dataset as a potential cause?  If not I'm
>>>> very interested in the details.
>>>
>>>
>>> Please, tell me how to respond to this question!
>>>
>>
>> Sorry, I wasn't clear.  It sounds like you've got two very different
>> copy speeds on two possibly different data sets.  The problem will be
>> significantly easier to debug and discuss if you boot a 2.4.19 + data
>> logging kernel, do the test, and then do the test again on a 2.4.21-pre
>> data logging kernel.
>>
>> I know this is a pain, I want to reduce the number of variables in
>> place.
>>
> 
> Yes. That all is why I asked. I'll try to find and retest the old things 
> tomorrow night. The new setup will go fine with 2.4.21-pre5 ? Or is this 
> one more step of irritation? Mmm. Or should I wait for the patches you 
> announce below?
> 

Hi, and sorry for the delay.

So I copied over the same dataset several times. ;-)

The times measured include the umounts of both partitions.
The test dataset was freshly copied to the freshly created
  test FS under the same kernel version before timing.
This time I took the times of two subsequent runs.
The test script does only simple "cp -a ..."

/dev/hda11    5076348   3762908   1313420  75% /mnt/alpha
/dev/hdd11    5076348   3762908   1313440  75% /mnt/beta

2.4.21-pre5 data=ordered [A]
       first run  second run  diff  average    rate
real  9m05.375s   9m23.272s  3.2%  9m14.324s  6.63MB/s
user  0m01.440s   0m01.730s
sys   1m11.640s   1m08.870s

2.4.19 data=ordered [B]
       first run  second run  diff  average    rate
real  7m46.287s   7m34.031s  2.7%  7m40.159s  7.99MB/s
user  0m01.520s   0m01.580s
sys   1m21.920s   1m21.300s


-> I didn't know that subsequent runs on the same conditions
    differ by ~3% ( -> So don't take my statement of the 2.5%
    difference in my previous mail for that serious as I
    only timed one run that time)

-> The throughput rate under 2.4.19 is about 20% higher
    than under 2.4.21-pre5 with this dataset


I hope it's "visible" this time...  ;-))


Best regards,

Manuel



P.S.: Disk data distribution file is available if you want.


Below the list of used patches just to be complete...

[A] Patches for 2.4.21-pre5:
01.m02-akpm-b_journal_head-1.diff
02.m03-relocation-6.diff
03.m04-reiserfs-sync_fs-1.diff
04.m05-data-logging-36.namesys.diff
05.m06-logging-export.diff
06.m06-write_times.diff
07.m09.kinoded-9.diff
08.m11.inode-dirty-for-kinoded.ML.diff
09.iget5_locked_for_datalogging.namesys.noquota.diff
preempt-kernel-rml-2.4.21-pre5-pre1-0.patch

[B] Patches for 2.4.19:
01.change_help_wording.diff
02.change_menu_wording.diff
03.confusing_journal_replay_warning_removal.diff
05.hash_on_empty_fs-1.diff
07.m01-relocation-4.diff
08.m02-commit_super-9-relocation.diff
09.m03-data-logging-25.diff
10.m03-logging-export.diff
11.m04-write_times.diff
12.m05-search_reada-4.diff
preempt-kernel-rml-2.4.19-2.patch

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sat, 2003-03-08 at 09:13, Manuel Krause wrote:

> 
> Hi, and sorry for the delay.
> 
> So I copied over the same dataset several times. ;-)
> 
> The times measured include the umounts of both partitions.
> The test dataset was freshly copied to the freshly created
>   test FS under the same kernel version before timing.
> This time I took the times of two subsequent runs.
> The test script does only simple "cp -a ..."
> 

Thank you very much for the details.  Just to make sure I understand how
you are generating the numbers, could I please see the test script?

It sounds like you:

copy dataset to fresh fs
time copy dataset from freshfs sub1 to freshfs sub2?

is that right?

-chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 03/09/2003 05:39 PM, Chris Mason wrote:
> On Sat, 2003-03-08 at 09:13, Manuel Krause wrote:
> 
> 
>>Hi, and sorry for the delay.
>>
>>So I copied over the same dataset several times. ;-)
>>
>>The times measured include the umounts of both partitions.
>>The test dataset was freshly copied to the freshly created
>>  test FS under the same kernel version before timing.
>>This time I took the times of two subsequent runs.
>>The test script does only simple "cp -a ..."
>>
> 
> 
> Thank you very much for the details.  Just to make sure I understand how
> you are generating the numbers, could I please see the test script?
> 
> It sounds like you:
> 
> copy dataset to fresh fs
> time copy dataset from freshfs sub1 to freshfs sub2?
> 
> is that right?
> 

Hi!

What do you mean with "sub"? Sub-directories?


My procedure is the following:

->
mkreiserfs partition2 (second drive)

copy dataset from used partition1 to partition2
  (identical, yes)

mkreiserfs partition1 (first drive)

time (copy dataset from partition2 to partition1
                         + umount partitions 1 and 2)

<-

In fact that are two dumb scripts from years ago - one for the first 
direction and one for the second. The time command line is the following:
  time sh -c "cp -ax /mnt/alpha/. /mnt/beta/ ; umount /mnt/alpha ; 
umount /mnt/beta"

And again I may have forgotten to mention that I always mount my 
partitions with "notail":
  mount -t reiserfs -o noatime,notail,data=ordered /dev/hd... /mnt/...

Does this information make it more understandable?


Bye,

Manuel

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sun, 2003-03-09 at 12:07, Manuel Krause wrote:

> 
> My procedure is the following:
> 
> ->
> mkreiserfs partition2 (second drive)
> 
> copy dataset from used partition1 to partition2
>   (identical, yes)
> 
> mkreiserfs partition1 (first drive)
> 
> time (copy dataset from partition2 to partition1
>                          + umount partitions 1 and 2)
> 
> <-
> 
> In fact that are two dumb scripts from years ago - one for the first 
> direction and one for the second. The time command line is the following:
>   time sh -c "cp -ax /mnt/alpha/. /mnt/beta/ ; umount /mnt/alpha ; 
> umount /mnt/beta"
> 
> And again I may have forgotten to mention that I always mount my 
> partitions with "notail":
>   mount -t reiserfs -o noatime,notail,data=ordered /dev/hd... /mnt/...
> 
> Does this information make it more understandable?

Very much, thanks.  Thanks to some work Jeff recently did, my current
thought is the slow down is related to the block allocator, and the
search_by_reada patch.  Jeff and I are working up numbers to see if this
is right, more details on monday.

-chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 03/09/2003 06:24 PM, Chris Mason wrote:
> On Sun, 2003-03-09 at 12:07, Manuel Krause wrote:
[...]
>>
>>Does this information make it more understandable?
> 
> 
> Very much, thanks.  Thanks to some work Jeff recently did, my current
> thought is the slow down is related to the block allocator, and the
> search_by_reada patch.  Jeff and I are working up numbers to see if this
> is right, more details on monday.
> 

I hope you don't blame the search_by_reada patch itself for a slowdown. 
There is a 05-search_reada-4.diff in my said 2.4.19 setup. Or didn't you 
integrate that patch in your other patches since then - and excluded it?

Bye,

Manuel

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sun, 2003-03-09 at 12:39, Manuel Krause wrote:
> > Very much, thanks.  Thanks to some work Jeff recently did, my current
> > thought is the slow down is related to the block allocator, and the
> > search_by_reada patch.  Jeff and I are working up numbers to see if this
> > is right, more details on monday.
> > 
> 
> I hope you don't blame the search_by_reada patch itself for a slowdown. 
> There is a 05-search_reada-4.diff in my said 2.4.19 setup. Or didn't you 
> integrate that patch in your other patches since then - and excluded it?

Just the opposite, I think the search_reada patch helps, but not with
the default allocator in 2.4.21.  The metadata seems to be spread too
far apart there.  I promise, more details on monday after we get some
additional numbers to back things up ;-)

-chris

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sun, 2003-03-09 at 12:39, Manuel Krause wrote:

> I hope you don't blame the search_by_reada patch itself for a slowdown. 
> There is a 05-search_reada-4.diff in my said 2.4.19 setup. Or didn't you 
> integrate that patch in your other patches since then - and excluded it?

Ok, our results so far are really interesting, they basically show the
default allocator in 2.4.20 & 2.4.21 fragmenting badly as the number of
concurrent writers increases.

But, you've only got one writer, so I need to do a better job of
modeling your data set.  Could you please send along the file size
distribution?

-chris

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

[ slow down between 2.419 data logging and 2.4.21-pre data logging ]

Ok, could you please apply the search_reada-4 patch I sent and do your
benchmark again on 2.4.21-preX with these mount options:

mount -o alloc=old_way:concentrating_formatted_nodes,notail

It's about 10% faster here, but my dataset is slightly different.

-chris

Re: [PATCH] new data logging and quota patches available

[Oleg Drokin]

Hello!

On Tue, Mar 11, 2003 at 03:46:18PM -0500, Chris Mason wrote:
> [ slow down between 2.419 data logging and 2.4.21-pre data logging ]
> Ok, could you please apply the search_reada-4 patch I sent and do your

> It's about 10% faster here, but my dataset is slightly different.

Hm. Last time we had conversation with you on this patch (a year ago?
may be even more), you said this patch makes no difference to you at all.
And we thought this is because your system is all-scsi, and scsi appears
to be more intelligent about caching/readahead.
Have you switched to IDE already? ;)

Bye,
    Oleg

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Wed, 2003-03-12 at 02:38, Oleg Drokin wrote:
> Hello!
> 
> On Tue, Mar 11, 2003 at 03:46:18PM -0500, Chris Mason wrote:
> > [ slow down between 2.419 data logging and 2.4.21-pre data logging ]
> > Ok, could you please apply the search_reada-4 patch I sent and do your
> 
> > It's about 10% faster here, but my dataset is slightly different.
> 
> Hm. Last time we had conversation with you on this patch (a year ago?
> may be even more), you said this patch makes no difference to you at all.
> And we thought this is because your system is all-scsi, and scsi appears
> to be more intelligent about caching/readahead.
> Have you switched to IDE already? ;)

No, that was on a scsi drive ;-)  Which patch, the search_reada?  Or
mounting with the border?

search_reada doesn't make much difference without the border, and even
then it really only helps on long sequential reads.

-chris

Re: [PATCH] new data logging and quota patches available

[Oleg Drokin]

Hello!

On Wed, Mar 12, 2003 at 08:22:30AM -0500, Chris Mason wrote:
> > > [ slow down between 2.419 data logging and 2.4.21-pre data logging ]
> > > Ok, could you please apply the search_reada-4 patch I sent and do your
> > > It's about 10% faster here, but my dataset is slightly different.
> > Hm. Last time we had conversation with you on this patch (a year ago?
> > may be even more), you said this patch makes no difference to you at all.
> > And we thought this is because your system is all-scsi, and scsi appears
> > to be more intelligent about caching/readahead.
> > Have you switched to IDE already? ;)
> No, that was on a scsi drive ;-)  Which patch, the search_reada?  Or

Hm. Something have changed then. I can dig the email, if you want.

> mounting with the border?

search_reada patch.

> search_reada doesn't make much difference without the border, and even

Sure.

> then it really only helps on long sequential reads.

That's true, but it made no difference for you on SCSI for sequentional reads too.

Bye,
    Oleg

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Wed, 2003-03-12 at 08:28, Oleg Drokin wrote:

> 
> > then it really only helps on long sequential reads.
> 
> That's true, but it made no difference for you on SCSI for sequentional reads too.

That's true, we were comparing your IDE drive results with my scsi for
reading a single big file.  It probably didn't help for the single big
file because of the page cache readahead triggering parallel tree
searches.

But Manuel's test is different, it's doing a long sequential read of
multiple files.  The page cache code only does readahead on one file at
a time, reading fileA won't trigger a parallel read of the indirect
items for fileB.  search_reada will work better in that case, since it
will grab indirect items for fileB.

Also, it will do readahead on the directory items, probably making
directory scanning faster.

-chris

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 03/11/2003 09:46 PM, Chris Mason wrote:
> [ slow down between 2.419 data logging and 2.4.21-pre data logging ]
> 
> Ok, could you please apply the search_reada-4 patch I sent and do your
> benchmark again on 2.4.21-preX with these mount options:
> 
> mount -o alloc=old_way:concentrating_formatted_nodes,notail
> 
> It's about 10% faster here, but my dataset is slightly different.
> 

I did it and also tested without the alloc mount option from above.

Here is the overview including the "old" things to refer to. The dataset 
didn't change in the meantime.

(old)
2.4.21-pre5 data=ordered                              [A]
       first run  second run  diff  average    rate
real  9m05.375s   9m23.272s  3.2%  9m14.324s  6.63MB/s
user  0m01.440s   0m01.730s
sys   1m11.640s   1m08.870s

2.4.19 data=ordered                                   [B]
       first run  second run  diff  average    rate
real  7m46.287s   7m34.031s  2.7%  7m40.159s  7.99MB/s
user  0m01.520s   0m01.580s
sys   1m21.920s   1m21.300s


(new)
2.4.21-pre5 [A] + search_reada-4 +
mount -o noatime,alloc=old_way:concentrating_formatted_nodes,notail [C]

       first run second run  diff   average    rate
real  7m59.631s  7m57.167s  0.52%  7m58.399s  7.68MB/s
user  0m01.580s  0m01.250s
sys   1m11.200s  1m09.520s


2.4.21-pre5 [A] + search_reada-4 without special mount options
(is: mount -o noatime,notail,data=ordered)                          [D]

       first run second run  diff   average    rate
real  7m57.365s  7m55.638s  0.36%  7m56.502s  7.71MB/s
user  0m01.510s  0m01.570s
sys   1m07.730s  1m08.020s


* What means a performance increase of 2.4.21-pre5 with [D] vs. without
    search_reada-4 [A] by  ~16%  !!!
* The difference of the special mount option [C] against my default [D]
   is about 0.3%  ;-))
* With search_reada-4  2.4.21-pre5 [D] only looses ~3.6% throughput rate
    against 2.4.19 [B]
  ...in this test.


Do you think the group-alloc-4 patch would increase throughput with this 
test?

Thank you for your great work,

Manuel

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Wed, 2003-03-12 at 09:43, Manuel Krause wrote:

> * What means a performance increase of 2.4.21-pre5 with [D] vs. without
>     search_reada-4 [A] by  ~16%  !!!
> * The difference of the special mount option [C] against my default [D]
>    is about 0.3%  ;-))
> * With search_reada-4  2.4.21-pre5 [D] only looses ~3.6% throughput rate
>     against 2.4.19 [B]
>   ...in this test.
> 

Outstanding, things are starting to make sense.

> 
> Do you think the group-alloc-4 patch would increase throughput with this 
> test?
> 

No, it will make this test slower (this is the default 2.4.21
allocator's best case)

> Thank you for your great work,

Thank you for all the testing and analysis, it really helps us.

-chris

Re: [PATCH] new data logging and quota patches available

[Hans Reiser]

Manuel Krause wrote:

> On 03/11/2003 09:46 PM, Chris Mason wrote:
>
>> [ slow down between 2.419 data logging and 2.4.21-pre data logging ]
>>
>> Ok, could you please apply the search_reada-4 patch I sent and do your
>> benchmark again on 2.4.21-preX with these mount options:
>>
>> mount -o alloc=old_way:concentrating_formatted_nodes,notail
>>
>> It's about 10% faster here, but my dataset is slightly different.
>>
>
> I did it and also tested without the alloc mount option from above.
>
> Here is the overview including the "old" things to refer to. The 
> dataset didn't change in the meantime.
>
> (old)
> 2.4.21-pre5 data=ordered                              [A]
>       first run  second run  diff  average    rate
> real  9m05.375s   9m23.272s  3.2%  9m14.324s  6.63MB/s
> user  0m01.440s   0m01.730s
> sys   1m11.640s   1m08.870s
>
> 2.4.19 data=ordered                                   [B]
>       first run  second run  diff  average    rate
> real  7m46.287s   7m34.031s  2.7%  7m40.159s  7.99MB/s
> user  0m01.520s   0m01.580s
> sys   1m21.920s   1m21.300s
>
>
> (new)
> 2.4.21-pre5 [A] + search_reada-4 +
> mount -o noatime,alloc=old_way:concentrating_formatted_nodes,notail [C]
>
>       first run second run  diff   average    rate
> real  7m59.631s  7m57.167s  0.52%  7m58.399s  7.68MB/s
> user  0m01.580s  0m01.250s
> sys   1m11.200s  1m09.520s
>
>
> 2.4.21-pre5 [A] + search_reada-4 without special mount options
> (is: mount -o noatime,notail,data=ordered)                          [D]
>
>       first run second run  diff   average    rate
> real  7m57.365s  7m55.638s  0.36%  7m56.502s  7.71MB/s
> user  0m01.510s  0m01.570s
> sys   1m07.730s  1m08.020s
>
>
> * What means a performance increase of 2.4.21-pre5 with [D] vs. without
>    search_reada-4 [A] by  ~16%  !!!
> * The difference of the special mount option [C] against my default [D]
>   is about 0.3%  ;-))
> * With search_reada-4  2.4.21-pre5 [D] only looses ~3.6% throughput rate
>    against 2.4.19 [B]
>  ...in this test.
>
>
> Do you think the group-alloc-4 patch would increase throughput with 
> this test?
>
> Thank you for your great work,
>
> Manuel
>
>
>
>
>
Can you state what test is being performed, what the hardware is, etc.  
In general, we need benchmarks that can be collected onto webpages and 
reviewed a year or two later when we can barely remember why we did or 
did not do some design decision.

-- 
Hans

Re: [PATCH] new data logging and quota patches available

[Manuel Krause]

On 03/12/2003 08:25 PM, Hans Reiser wrote:
> Manuel Krause wrote:
> 
>> On 03/11/2003 09:46 PM, Chris Mason wrote:
>>
>>> [ slow down between 2.419 data logging and 2.4.21-pre data logging ]
>>>
>>> Ok, could you please apply the search_reada-4 patch I sent and do your
>>> benchmark again on 2.4.21-preX with these mount options:
>>>
>>> mount -o alloc=old_way:concentrating_formatted_nodes,notail
>>>
>>> It's about 10% faster here, but my dataset is slightly different.
>>>
>>
[ timing data ]
>>
>> * What means a performance increase of 2.4.21-pre5 with [D] vs. without
>>    search_reada-4 [A] by  ~16%  !!!
>> * The difference of the special mount option [C] against my default [D]
>>   is about 0.3%  ;-))
>> * With search_reada-4  2.4.21-pre5 [D] only looses ~3.6% throughput rate
>>    against 2.4.19 [B]
>>  ...in this test.
>>
>>
>> Do you think the group-alloc-4 patch would increase throughput with 
>> this test?
>>
>> Thank you for your great work,
>>
>> Manuel
>>
>>
> Can you state what test is being performed, what the hardware is, etc.  
> In general, we need benchmarks that can be collected onto webpages and 
> reviewed a year or two later when we can barely remember why we did or 
> did not do some design decision.
> 

Hi Hans,

I think the way I "test" (is: simple timed backup-copy / defrag cycle) 
was effectively clarified in this thread.

Last autumn I wanted to add the desired system info to the ReiserFS 
regarding parts of my homepage but skipped it at some point simply 
lacking time...

You'll receive a compiled file of that info you might want, with the 
results I snipped above and the file size distribution data in the 
followup mail in private so you could give it to the persons of your 
team who collect it for the archives and/or prepare it for publishing. 
If you need differnt or more information, please, let me know.

Best regards,

Manuel

Re: [PATCH] new data logging and quota patches available

[Marc-Christian Petersen]

On Monday 24 February 2003 02:42, Manuel Krause wrote:

Hi Manuel,

> I see a performance decrease with my copies in data=ordered mode by
> about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19%
> decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing
> this? Partly I can blame it on the disk content distribution that
> changed. If I removed the kinoded related patches would the throughput
> rate increase?
I can confirm that this is also true for me. I am using the -aa data-logging 
36.

ciao, Marc

Re: [PATCH] new data logging and quota patches available

[Marc-Christian Petersen]

On Monday 24 February 2003 18:59, Marc-Christian Petersen wrote:

Hi again,

> > I see a performance decrease with my copies in data=ordered mode by
> > about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19%
> > decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing
> > this? Partly I can blame it on the disk content distribution that
> > changed. If I removed the kinoded related patches would the throughput
> > rate increase?
> I can confirm that this is also true for me. I am using the -aa
> data-logging 36.
grmpf, I forgot to add one thing here ;(

I just tried it with:

- time seq -f "%06.0f" 1 100000  | xargs touch

v35-aa took about 4 seconds to complete this.
v36-aa took about 6 seconds to compelte this.

Mounted with:
nosuid,nodev,noexec,noatime,nodiratime,notail,block-allocator=noborder

It is my SQUID cache directory.

ciao, Marc

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Mon, 2003-02-24 at 13:05, Marc-Christian Petersen wrote:

> I just tried it with:
> 
> - time seq -f "%06.0f" 1 100000  | xargs touch
> 
> v35-aa took about 4 seconds to complete this.
> v36-aa took about 6 seconds to compelte this.
> 
> Mounted with:
> nosuid,nodev,noexec,noatime,nodiratime,notail,block-allocator=noborder

For benchmarks that short, bdflush/kupdate can come in and screw the
results.  Also, when I create 100,000 files on my test box with these
mount parameters, it results in 3 large async transactions.

In data-logging-36, kreiserfsd will get those async commits to disk
sooner than in data-logging-35, which is the right thing to do because
it helps make sure later writes won't stall while we have to flush that
async transaction.

But for a very short benchmark that doesn't do more work later on, it
can seem slower.  If you want a more accurate picture, include the time
it takes to sync after creating all the files.

-chris

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Mon, 2003-02-24 at 12:59, Marc-Christian Petersen wrote:
> On Monday 24 February 2003 02:42, Manuel Krause wrote:
> 
> Hi Manuel,
> 
> > I see a performance decrease with my copies in data=ordered mode by
> > about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19%
> > decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing
> > this? Partly I can blame it on the disk content distribution that
> > changed. If I removed the kinoded related patches would the throughput
> > rate increase?
> I can confirm that this is also true for me. I am using the -aa data-logging 
> 36.

Sorry, you confirm that with kinoded related patches removed (and the
dirty_inode patch) things get faster?

Also, are you using kinoded-8 or kinoded-9?

-chris

Re: [PATCH] new data logging and quota patches available

[Marc-Christian Petersen]

On Monday 24 February 2003 19:08, Chris Mason wrote:

Hi Chris,

> > I can confirm that this is also true for me. I am using the -aa
> > data-logging 36.
> Sorry, you confirm that with kinoded related patches removed (and the
> dirty_inode patch) things get faster?
> Also, are you using kinoded-8 or kinoded-9?
Gna, sorry. There is no difference between kinoded-8 or kinoded-9 or w/o them.

It drops. See my other mail I just wrote to the list.

ciao, Marc

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sat, 2003-02-22 at 19:50, Manuel Krause wrote:

[ patch to leave inodes dirty instead of constantly logging them ]
> Hi Chris!
> 
> At least I'm not able to copy through my partition via "cp -a ..." with 
> the last supposed setup + preempt without quota ...
> 
> When umounting the destination partition it says "device is busy".
> 
> I don't know what is gone for now on my repository partition, but I know 
> from earlier times this should not happen.

A side not about this optimization.  It only affects inode updates for
mtime/ctime/atime, critical things like updating i_size or the block
count or other stat data operations are still logged every time, and
always logged in the same transaction as the critical metadata update.

In other words, this patch restores reiserfs to its 2.2.x inode update
scheme.

-chris

Re: [PATCH] new data logging and quota patches available

[Oleg Drokin]

Hello!

On Fri, Feb 21, 2003 at 06:32:11PM -0500, Chris Mason wrote:

> ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be
> updated with a new set of data logging and quota patches against
> 2.4.21-pre4
> The data logging code is updated with another set of io stalling fixes,
> they should improve performance of data=ordered and data=writeback by
> being smarter about forcing commits under heavy write load and kicking
> kreiserfsd.
> Treat these with care, they've gotten a ton of testing under the suse
> kernel, but the port to vanilla was just done today.
> The quota patches include a fix for incorrect sd_block counts on
> symlinks.

Replacement 05-data-logging-36.diff.gz file that applies to 2.4.21-pre4-ac5
is available at
ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-data-logging-36-ac5.diff.gz
It compiles, boots, survives my (simple) testing. (writing this email
from patched 2.4.21-pre4-ac5, too). Quota works. symlinks are now have correct
blocks count too
The reason for rejects is mostly DIRECTIO fix that also went into current
bk snapshot, so probably it will apply to Marcelo's bk tree, too.
Chris: Is it intended that directio only works on data=writeback
mounted filesystems?

Also following README file diff should be considered:

--- README.orig	Sat Feb 22 16:44:34 2003
+++ README	Sat Feb 22 16:44:49 2003
@@ -28,7 +28,7 @@
 
 These add reiserfs quota support
 
-07-quota-v2-2.4.21.diff.gz
+07-quota-v2-2.4.21.diff.gz # you don't need this on -ac, too
 08-reiserfs-quota-26.diff.gz
 09-kinoded-8.diff.gz
 

Re: [PATCH] new data logging and quota patches available

[Chris Mason]

On Sat, 2003-02-22 at 08:54, Oleg Drokin wrote:

> Replacement 05-data-logging-36.diff.gz file that applies to 2.4.21-pre4-ac5
> is available at
> ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-data-logging-36-ac5.diff.gz

Thanks Oleg.

> It compiles, boots, survives my (simple) testing. (writing this email
> from patched 2.4.21-pre4-ac5, too). Quota works. symlinks are now have correct
> blocks count too
> The reason for rejects is mostly DIRECTIO fix that also went into current
> bk snapshot, so probably it will apply to Marcelo's bk tree, too.
>
> Chris: Is it intended that directio only works on data=writeback
> mounted filesystems?
> 

Yes, the way ordered writes work is the buffers are put onto a per
transaction list that gets flushed before the commit.  Since buffers can
only be on one list, this means they don't get onto the list of buffers
for that particular inode, and that makes it difficult to make sure all
pending io to the file is finished before allowing the directio to
proceed (just like the tail alias bug).

The only way to do it is by forcing a commit before the directio, which
would be horribly slow so I've disabled the data=ordered o_direct
support.  The real solution is to allocate a data structure and point to
it from the private journal pointer in the bh.  That requires some other
changes and other performance stuff is more important right now.

> Also following README file diff should be considered:
> 

Thanks, I'm uploading a few more optimizations shortly, I'll include
this change.

-chris

Re: [PATCH] new data logging and quota patches available

[Ookhoi]

Hi Chris and Oleg,

Oleg Drokin wrote (ao):
> On Fri, Feb 21, 2003 at 06:32:11PM -0500, Chris Mason wrote:
> 
> > ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be
> > updated with a new set of data logging and quota patches against
> > 2.4.21-pre4

[cut]

> Replacement 05-data-logging-36.diff.gz file that applies to
> 2.4.21-pre4-ac5 is available at
> ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-data-logging-36-ac5.diff.gz
> It compiles, boots, survives my (simple) testing. (writing this email
> from patched 2.4.21-pre4-ac5, too). Quota works. symlinks are now have
> correct blocks count too

Light testing on 2.4.21-pre4-ac6 with the following patches is 100%
succesful:

(patch-2.4.21-pre4)
(patch-2.4.21-pre4-ac6)
03-relocation-6.diff 
04-reiserfs-sync_fs-1.diff 
05-data-logging-36-ac5.diff 
06-logging-export.diff 
08-reiserfs-quota-26.diff
09-kinoded-8.diff
10-reiserfs-quota-link-fix.diff

They apply with zero rejects.

Thanks a lot!!

If you have specific things you want to have tested, please let me know.
Quota is a It Works For Me(tm), but only very light testing.