Re: Question regarding to store file system metadata in database

[Matti Aarnio <matti.aarnio@zmailer.org>]

On Mon, Mar 20, 2006 at 02:36:51PM -0500, Xin Zhao wrote:
> 
> OK. Now I have more experimental results.
> 
> After excluding the cost of reading file list and do stat(), the
> insertion rate becomes 587/sec, instead of 300/sec. The query rate is
> 2137/sec. I am runing mysql 4.1.11. FC4, 2.8G CPU and 1G mem.
> 
> 2137/sec seems to be good enough to handle pathname to inode
> resolving.  Anyone has some statistics how many file open in a busy
> file system?
> 
> Xin

What is wrong in here, I think, is your pre-set assumption, that
using proper modern database things will be faster.   Yes, perhaps
they will, under some specific conditions.

Like Gene Amdahl so long ago did point out, optimizing something
that forms 1% of the load will speed things up at most that 1%.

Could you instrument directory management primitive operations 
accounting ?  How many directory inserts/removes/lookups per
mounted filesystem (or entire system), including  dnames -cache
operations (they are already instrumented, I think) are used in
normal system operations ?

If your system behaviour shows more than 1% of other than lookups,
try to find out _why_ is that.

So far Linux optimizes filesystem directory reads to maximum.



Long ago I had a problem, where I needed insertion into an application
specific database from data origination system -- I needed also fast
batch replication from one dataset copy to another.  Doing hash keying
made insert _slow_.  Doing btree indexing and inserting in key-order
made things fast.   Not flushing database at every insert made it almost
linearly faster by the flush interval.  Not flushing the database except
at batch end made it maximally fast -- around 100 000 inserts per second,
but it had to be pre-sorted data.  (This was single SCSI-disk back in
1996.)  We had requirement to do batch insert as fast as possible,
similarly for batch replication that was used for maintenance, and
a ten-thousand-fold speedup was well worth the added complexities
in the software.

That database had also about 4 sigma "read-only" property.


 /Matti Aarnio