dbench regression in 2.6

dbench regression in 2.6

[Steven Pratt]

Is anyone looing into the dbench multitheaded regression in 2.6 that I 
reported here a couple of weeks ago?  I don't see any change in the 
latest trees.

http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png

Steve

Re: dbench regression in 2.6

[Mike Fedyk]

On Wed, Sep 03, 2003 at 01:46:13PM -0500, Steven Pratt wrote:
> Is anyone looing into the dbench multitheaded regression in 2.6 that I 
> reported here a couple of weeks ago?  I don't see any change in the 
> latest trees.
> 
> http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
> 

If it's only in dbench, it may not be a problem per say.  Dbench does best
when there is less fairness.

Check the latest posts of contest.  There are about 7 improvements with
about 3 regressions.  Dbench regression by itself doesn't really mean a bad
thing happened.

Re: dbench regression in 2.6

[Mike Fedyk]

On Wed, Sep 03, 2003 at 01:46:13PM -0500, Steven Pratt wrote:
> Is anyone looing into the dbench multitheaded regression in 2.6 that I 
> reported here a couple of weeks ago?  I don't see any change in the 
> latest trees.
> 
> http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
> 

 host ausgsa.ibm.com
 ausgsa.ibm.com does not exist, try again
 

Re: dbench regression in 2.6

[Vladimir Saveliev]

Hi

On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
> Steven Pratt wrote:
> 
> > Is anyone looing into the dbench multitheaded regression in 2.6 that I 
> > reported here a couple of weeks ago? 

Sorry, I was unable to find your report in mail archives. Could you, please, 
remind what is the problem?

 I don't see any change in the 
> > latest trees.
> >
> > 
http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
> >

Is this address right?
> ping ausgsa.ibm.com
ping: unknown host ausgsa.ibm.com
>

Thanks,
vs


> > Steve
> >
> >
> >
> It does rather look like we have a systemic failure to track issues.  I 
> am not happy about this.   We will discuss this privately for a bit and 
> then get back to you.
> 
> -- 
> Hans
> 
> 
> 
> 

Re: dbench regression in 2.6

[Steven Pratt]

Mike Fedyk wrote:

>On Wed, Sep 03, 2003 at 01:46:13PM -0500, Steven Pratt wrote:
>  
>
>>Is anyone looing into the dbench multitheaded regression in 2.6 that I 
>>reported here a couple of weeks ago?  I don't see any change in the 
>>latest trees.
>>
>>http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
>>
>>    
>>
>
> host ausgsa.ibm.com
> ausgsa.ibm.com does not exist, try again
> 
>
Sorry, my bad.  This is the internal site where we generate the data.  
The public site is:

http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.16.png

Steve

Re: dbench regression in 2.6

[Steven Pratt]

Vladimir Saveliev wrote:

>Hi
>
>On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
>  
>
>>Steven Pratt wrote:
>>
>>    
>>
>>>Is anyone looking into the dbench multitheaded regression in 2.6 that I 
>>>reported here a couple of weeks ago? 
>>>      
>>>
>
>Sorry, I was unable to find your report in mail archives. Could you, please, 
>remind what is the problem?
>
> I don't see any change in the 
>  
>
>>>latest trees.
>>>
>>>      
>>>
The problem showed up in 2.6.0-test1, and every kernel since.  The 
problem is that from 2.5.65 (earliest data I have) to 2.5.69 dbench on 
reiserfs with 16 clients got a throughput of ~200MB/sec.  In 2.5.70 
through 2.5.75 the score went up to ~280MB/sec, a good thing.  But in 
2.6.0-test1 the throughput dropped to below 50 MB/sec.  Since then it 
has risen slightly but still at only around 60MB/sec.  Single client 
dbench showed similar, although much less dramatic changes.  No other 
file system exhibited a change in the 2.6.0-test1 kernel so this seems 
to be unique to reiserfs.


>>>      
>>>
>http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
>  
>
>
>Is this address right?
>
No, my bad.  Right address is 
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.16.png
see also
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.1.png

Also if you go to
http://ltcperf.ncsa.uiuc.edu/data/2.6.0-test1/2.5.75-vs-2.6.0-test1/index.html
and select the results links related to dbench resierfs you can find 
kernel profile, sar data and lots of other system information which may 
help in isolating the problem.

Steve

Re: dbench regression in 2.6

[Mike Fedyk]

On Thu, Sep 04, 2003 at 09:23:42AM -0500, Steven Pratt wrote:
> Vladimir Saveliev wrote:
> 
> >Hi
> >
> >On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
> > 
> >
> >>Steven Pratt wrote:
> >>
> >>   
> >>
> >>>Is anyone looking into the dbench multitheaded regression in 2.6 that I 
> >>>reported here a couple of weeks ago? 
> >>>     
> >>>
> >
> >Sorry, I was unable to find your report in mail archives. Could you, 
> >please, remind what is the problem?
> >
> >I don't see any change in the 
> > 
> >
> >>>latest trees.
> >>>
> >>>     
> >>>
> The problem showed up in 2.6.0-test1, and every kernel since.  The 
> problem is that from 2.5.65 (earliest data I have) to 2.5.69 dbench on 
> reiserfs with 16 clients got a throughput of ~200MB/sec.  In 2.5.70 
> through 2.5.75 the score went up to ~280MB/sec, a good thing.  But in 
> 2.6.0-test1 the throughput dropped to below 50 MB/sec.  Since then it 
> has risen slightly but still at only around 60MB/sec.  Single client 
> dbench showed similar, although much less dramatic changes.  No other 
> file system exhibited a change in the 2.6.0-test1 kernel so this seems 
> to be unique to reiserfs.

Did anything change in the testing at all?  Can you try again with your
current software versions on 2.5.75, and reproduce the higher speeds?

Check the diffstat for the test1 release and see if there are any reiserfs
changes...

Re: dbench regression in 2.6

[Steven Pratt]

Mike Fedyk wrote:

>On Thu, Sep 04, 2003 at 09:23:42AM -0500, Steven Pratt wrote:
>  
>
>>Vladimir Saveliev wrote:
>>
>>    
>>
>>>Hi
>>>
>>>On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
>>>
>>>      
>>>
>>>>Steven Pratt wrote:
>>>>
>>>>        
>>>>
>>>>>Is anyone looking into the dbench multitheaded regression in 2.6 that I 
>>>>>reported here a couple of weeks ago? 
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>Sorry, I was unable to find your report in mail archives. Could you, 
>>>please, remind what is the problem?
>>>
>>>I don't see any change in the 
>>>
>>>      
>>>
>>>>>latest trees.
>>>>>
>>>>>    
>>>>>
>>>>>          
>>>>>
>>The problem showed up in 2.6.0-test1, and every kernel since.  The 
>>problem is that from 2.5.65 (earliest data I have) to 2.5.69 dbench on 
>>reiserfs with 16 clients got a throughput of ~200MB/sec.  In 2.5.70 
>>through 2.5.75 the score went up to ~280MB/sec, a good thing.  But in 
>>2.6.0-test1 the throughput dropped to below 50 MB/sec.  Since then it 
>>has risen slightly but still at only around 60MB/sec.  Single client 
>>dbench showed similar, although much less dramatic changes.  No other 
>>file system exhibited a change in the 2.6.0-test1 kernel so this seems 
>>to be unique to reiserfs.
>>    
>>
>
>Did anything change in the testing at all?  Can you try again with your
>current software versions on 2.5.75, and reproduce the higher speeds?
>  
>
No, nothing changed in the test setup.  Same dbench, same disk, same 
distro, just different kernel. 

>Check the diffstat for the test1 release and see if there are any reiserfs
>changes...
>
I believe that Oleg already said there were no changes that look like 
they would have caused this.  More likely the behavior of something 
reiser is using changed which is having a bad effect on resierfs.

Steve

Re: dbench regression in 2.6

[Hans Reiser]

Steven Pratt wrote:

> Mike Fedyk wrote:
>
>> On Thu, Sep 04, 2003 at 09:23:42AM -0500, Steven Pratt wrote:
>>  
>>
>>> Vladimir Saveliev wrote:
>>>
>>>   
>>>
>>>> Hi
>>>>
>>>> On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
>>>>
>>>>     
>>>>
>>>>> Steven Pratt wrote:
>>>>>
>>>>>       
>>>>>
>>>>>> Is anyone looking into the dbench multitheaded regression in 2.6 
>>>>>> that I reported here a couple of weeks ago?   
>>>>>>         
>>>>>
>>>> Sorry, I was unable to find your report in mail archives. Could 
>>>> you, please, remind what is the problem?
>>>>
>>>> I don't see any change in the
>>>>     
>>>>
>>>>>> latest trees.
>>>>>>
>>>>>>   
>>>>>>         
>>>>>
>>> The problem showed up in 2.6.0-test1, and every kernel since.  The 
>>> problem is that from 2.5.65 (earliest data I have) to 2.5.69 dbench 
>>> on reiserfs with 16 clients got a throughput of ~200MB/sec.  In 
>>> 2.5.70 through 2.5.75 the score went up to ~280MB/sec, a good 
>>> thing.  But in 2.6.0-test1 the throughput dropped to below 50 
>>> MB/sec.  Since then it has risen slightly but still at only around 
>>> 60MB/sec.  Single client dbench showed similar, although much less 
>>> dramatic changes.  No other file system exhibited a change in the 
>>> 2.6.0-test1 kernel so this seems to be unique to reiserfs.
>>>   
>>
>>
>> Did anything change in the testing at all?  Can you try again with your
>> current software versions on 2.5.75, and reproduce the higher speeds?
>>  
>>
> No, nothing changed in the test setup.  Same dbench, same disk, same 
> distro, just different kernel.
>
>> Check the diffstat for the test1 release and see if there are any 
>> reiserfs
>> changes...
>>
> I believe that Oleg already said there were no changes that look like 
> they would have caused this.  More likely the behavior of something 
> reiser is using changed which is having a bad effect on resierfs.
>
> Steve
>
>
>
Vs discovered we are sleeping more in test0, and he is debugging the 
sleepometer so he can find out where (2.6.0 broke the sleepometer it seems).

-- 
Hans

Re: dbench regression in 2.6

[Vladimir Saveliev]

Hi

On Thursday 04 September 2003 18:23, Steven Pratt wrote:
> Vladimir Saveliev wrote:
> 
> >Hi
> >
> >On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
> >  
> >
> >>Steven Pratt wrote:
> >>
> >>    
> >>
> >>>Is anyone looking into the dbench multitheaded regression in 2.6 that I 
> >>>reported here a couple of weeks ago? 
> >>>      
> >>>
> >
> >Sorry, I was unable to find your report in mail archives. Could you, 
please, 
> >remind what is the problem?
> >
> > I don't see any change in the 
> >  
> >
> >>>latest trees.
> >>>
> >>>      
> >>>
> The problem showed up in 2.6.0-test1, and every kernel since.  The 
> problem is that from 2.5.65 (earliest data I have) to 2.5.69 dbench on 
> reiserfs with 16 clients got a throughput of ~200MB/sec.  In 2.5.70 
> through 2.5.75 the score went up to ~280MB/sec, a good thing.  But in 
> 2.6.0-test1 the throughput dropped to below 50 MB/sec.  Since then it 
> has risen slightly but still at only around 60MB/sec.  Single client 
> dbench showed similar, although much less dramatic changes.  No other 
> file system exhibited a change in the 2.6.0-test1 kernel so this seems 
> to be unique to reiserfs.
> 
> 
> >>>      
> >>>
> 
>http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
> >  
> >
> >
> >Is this address right?
> >
> No, my bad.  Right address is 
> 
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.16.png
> see also
> 
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.1.png
> 
> Also if you go to
> 
http://ltcperf.ncsa.uiuc.edu/data/2.6.0-test1/2.5.75-vs-2.6.0-test1/index.html
> and select the results links related to dbench resierfs you can find 
> kernel profile, sar data and lots of other system information which may 
> help in isolating the problem.
> 
> Steve
> 

Unfortunately, I was unable to reproduce your result. On our test machine (4 
Xeons 1gb ram) results shown by reiserfs on 2.5.75 and 2.6.0-test1 are 
comparable.

So, would be you so kind to help us to track this issue? If yes, could you , 
please, describe your test in more details.
0. what is version of dbench you used: 1.2, 1.3 or 2.0?
1. how many time did you run dbench 16?
2. when did you mkfs, etc, did you reboot after mkfs?
3. how much ram does machine which was running dbench have?
4. could you please try to rerun benchmark with SMP=off, and with 2 and 4 
cpus?
5. could you also run dbench 64 and dbench 128 for reiserfs 2.5.75 and 
2.6.0-test1?

Thanks,
vs




> 
> 
> 

Re: dbench regression in 2.6

[Steven Pratt]

Vladimir Saveliev wrote:

>Hi
>
>On Thursday 04 September 2003 18:23, Steven Pratt wrote:
>  
>
>>Vladimir Saveliev wrote:
>>
>>    
>>
>>>Hi
>>>
>>>On Wednesday 03 September 2003 23:04, Hans Reiser wrote:
>>> 
>>>
>>>      
>>>
>>>>Steven Pratt wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>Is anyone looking into the dbench multitheaded regression in 2.6 that I 
>>>>>reported here a couple of weeks ago? 
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>Sorry, I was unable to find your report in mail archives. Could you, 
>>>      
>>>
>please, 
>  
>
>>>remind what is the problem?
>>>
>>>I don't see any change in the 
>>> 
>>>      
>>>
>>>>>latest trees.
>>>>>
>>>>>     
>>>>>          
>>>>>
>>The problem showed up in 2.6.0-test1, and every kernel since.  The 
>>problem is that from 2.5.65 (earliest data I have) to 2.5.69 dbench on 
>>reiserfs with 16 clients got a throughput of ~200MB/sec.  In 2.5.70 
>>through 2.5.75 the score went up to ~280MB/sec, a good thing.  But in 
>>2.6.0-test1 the throughput dropped to below 50 MB/sec.  Since then it 
>>has risen slightly but still at only around 60MB/sec.  Single client 
>>dbench showed similar, although much less dramatic changes.  No other 
>>file system exhibited a change in the 2.6.0-test1 kernel so this seems 
>>to be unique to reiserfs
>>
>>http://ausgsa.ibm.com/projects/l/ltcperformance/2003benchmarks/regression/results/history-graphs/dbench.reiser.throughput.plot.16.png
>>    
>>
>>> 
>>>
>http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.16.png
>  
>
>>see also
>>
>>    
>>
>http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.reiser.throughput.plot.1.png
>  
>
>>Also if you go to
>>
>>    
>>
>http://ltcperf.ncsa.uiuc.edu/data/2.6.0-test1/2.5.75-vs-2.6.0-test1/index.html
>  
>
>>and select the results links related to dbench resierfs you can find 
>>kernel profile, sar data and lots of other system information which may 
>>help in isolating the problem.
>>
>>Steve
>>
>>    
>>
>
>Unfortunately, I was unable to reproduce your result. On our test machine (4 
>Xeons 1gb ram) results shown by reiserfs on 2.5.75 and 2.6.0-test1 are 
>comparable.
>
>So, would be you so kind to help us to track this issue? If yes, could you , 
>please, describe your test in more details.
>
Sure.

>0. what is version of dbench you used: 1.2, 1.3 or 2.0?
>
2.0

>1. how many time did you run dbench 16?
>
4 (for each client count, results are averaged)

>2. when did you mkfs, etc, did you reboot after mkfs?
>
No, but the fs is mkfsed between each run.

>3. how much ram does machine which was running dbench have?
>
8GB

>4. could you please try to rerun benchmark with SMP=off, and with 2 and 4 
>cpus?
>5. could you also run dbench 64 and dbench 128 for reiserfs 2.5.75 and 
>2.6.0-test1?
>  
>
Yes, but the machine is tied up at the moment, may take me a day or so 
to get more time on it.

>Thanks,
>vs
>
>  
>
Steve

Re: dbench regression in 2.6

[Steven Pratt]

Ok, first let me start with appologies as this turns out to be a wild 
goose chase.  It seems that I am running the version of reiserfsprogs 
that ships with SLES 8.0 (3.6.2) which has a hardcoded check for kernel 
versions 2.2, 2.4 and 2.5, but does not know about 2.6 and thus the 
mkfs.resiserfs is failing.  Since all of this testing is automated this 
failure was not caught, and the dbench numbers reported were actually 
for the last filesystem run on that drive, in this case XFS (which is 
why it looks so bad :-) ).  I have fixed the scripts to force the format 
to 3.6 and all seems to be well.  If I get the chance I will go backfill 
all of the numbers.  Sorry for the confusion but I run these regresion 
tests in my spare time and don't always have time to crawl through the 
output, although a simple look in the captured /proc/mounts, or even the 
fstype file I create in the benchmark directory would have saved us all 
a lot of time. 

Again, sorry for the confusion, keep up the good work.

Steve

Re: dbench regression in 2.6

[Hans Reiser]

Steven Pratt wrote:

> Ok, first let me start with appologies as this turns out to be a wild 
> goose chase.  It seems that I am running the version of reiserfsprogs 
> that ships with SLES 8.0 (3.6.2) which has a hardcoded check for 
> kernel versions 2.2, 2.4 and 2.5, but does not know about 2.6 and thus 
> the mkfs.resiserfs is failing.  Since all of this testing is automated 
> this failure was not caught, and the dbench numbers reported were 
> actually for the last filesystem run on that drive, in this case XFS 
> (which is why it looks so bad :-) ).  I have fixed the scripts to 
> force the format to 3.6 and all seems to be well.  If I get the chance 
> I will go backfill all of the numbers.  Sorry for the confusion but I 
> run these regresion tests in my spare time and don't always have time 
> to crawl through the output, although a simple look in the captured 
> /proc/mounts, or even the fstype file I create in the benchmark 
> directory would have saved us all a lot of time.
> Again, sorry for the confusion, keep up the good work.
>
> Steve
>
>
>
Steve, thanks a lot for making this effort.  I would be interested in 
your results.  Do you think it might be possible for us to test and 
profile reiser4 scalability on your 8-way?  We designed reiser4 to be 
highly scalable, but of course one always finds a few unexpected 
scalability issues when one tests for the first time.

-- 
Hans

Re: dbench regression in 2.6

[Steven Pratt]

Hans Reiser wrote:

> Steven Pratt wrote:
>
>> Ok, first let me start with appologies as this turns out to be a wild 
>> goose chase.  It seems that I am running the version of reiserfsprogs 
>> that ships with SLES 8.0 (3.6.2) which has a hardcoded check for 
>> kernel versions 2.2, 2.4 and 2.5, but does not know about 2.6 and 
>> thus the mkfs.resiserfs is failing.  Since all of this testing is 
>> automated this failure was not caught, and the dbench numbers 
>> reported were actually for the last filesystem run on that drive, in 
>> this case XFS (which is why it looks so bad :-) ).  I have fixed the 
>> scripts to force the format to 3.6 and all seems to be well.  If I 
>> get the chance I will go backfill all of the numbers.  Sorry for the 
>> confusion but I run these regresion tests in my spare time and don't 
>> always have time to crawl through the output, although a simple look 
>> in the captured /proc/mounts, or even the fstype file I create in the 
>> benchmark directory would have saved us all a lot of time.
>> Again, sorry for the confusion, keep up the good work.
>>
>> Steve
>>
>>
>>
> Steve, thanks a lot for making this effort.  I would be interested in 
> your results.  Do you think it might be possible for us to test and 
> profile reiser4 scalability on your 8-way?  We designed reiser4 to be 
> highly scalable, but of course one always finds a few unexpected 
> scalability issues when one tests for the first time.

Yes, I definitely plan on doing reiser4 benchmarking.  Just a matter of 
finding some time to do it.

Steve

Re: dbench regression in 2.6

[Nikita Danilov]

Steven Pratt writes:
 > Ok, first let me start with appologies as this turns out to be a wild 
 > goose chase.  It seems that I am running the version of reiserfsprogs 
 > that ships with SLES 8.0 (3.6.2) which has a hardcoded check for kernel 
 > versions 2.2, 2.4 and 2.5, but does not know about 2.6 and thus the 
 > mkfs.resiserfs is failing.  Since all of this testing is automated this 
 > failure was not caught, and the dbench numbers reported were actually 
 > for the last filesystem run on that drive, in this case XFS (which is 
 > why it looks so bad :-) ).  I have fixed the scripts to force the format 

It is a pity you found this so fast. I guess in a day or two Vladimir
would have determined and fixed the reason of this "regression" and sped
up reiserfs. :)

 > to 3.6 and all seems to be well.  If I get the chance I will go backfill 
 > all of the numbers.  Sorry for the confusion but I run these regresion 
 > tests in my spare time and don't always have time to crawl through the 
 > output, although a simple look in the captured /proc/mounts, or even the 
 > fstype file I create in the benchmark directory would have saved us all 
 > a lot of time. 
 > 
 > Again, sorry for the confusion, keep up the good work.
 > 
 > Steve
 > 

Nikita.