Re: [resent PATCH] Re: very slow parallel read performance

[Hans Reiser <reiser@namesys.com>]

Andrew Morton wrote:
> 
> Dieter NЭtzel wrote:
> >
> > ...
> > (dbench-1.1 32 clients)
> > ...
> > 640 MB PC100-2-2-2 SDRAM
> > ...
> > * readahead do not show dramatic differences
> > * killall -STOP kupdated DO
> 
> dbench is a poor tool for evaluating VM or filesystem performance.
> It deletes its own files.
> 
> If you want really good dbench numbers, you can simply install lots
> of RAM and tweak the bdflush parameters thusly:
> 
> 1: set nfract and nfract_sync really high, so you can use all your
>    RAM for buffering dirty data.
> 
> 2: Set the kupdate interval to 1000000000 to prevent kupdate from
>    kicking in.
> 
> And guess what?  You can perform an entire dbench run without
> touching the disk at all!  dbench deletes its own files, and
> they never hit disk.
> 
> It gets more complex - if you leave the bdflush parameters at
> default, and increase the number of dbench clients you'll reach
> a point where bdflush starts kicking in to reduce the amount
> of buffercache memory.  This slows the dbench clients down,
> so they have less opportunity to delete data before kupdate and
> bdflush write them out.  So the net effect is that the slower
> you go, the more I/O you end up doing - a *lot* more.  This slows
> you down further, which causes more I/O, etc.
> 
> dbench is not a benchmark.  It is really complex, it is really
> misleading.  It is a fine stress-tester though.
> 
> The original netbench test which dbench emulates has three phases:
> startup, run and cleanup.  Throughput numbers are only quoted for
> the "run" phase.  dbench is incomplete in that it reports on throughput
> for all three phases.  Apparently Tridge and friends are working on
> changing this, but it will still be the case that the entire test
> can be optimised away, and that it is subject to the regenerative
> feedback phenomenon described above.
> 
> For tuning and measuring fs and VM efficiency we need to user
> simpler, more specific tools.
> 
> -
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

I would encourage you to check out the reiser_fract_tree program, which is at
the heart of mongo.pl.  It generates lots of files using a fractal algorithm for
size and number per directory, and I think it reflects real world statistics
decently.  You can specify mean file size, max file size, mean nr of files per
directory, max nr files per directory, check it out..... 
www.namesys.com/benchmarks.html

It just generates file sets, which is fine for write performance testing like
you are doing.  Mongo.pl can do reads and copies and stuff using those file
sets.

Hans