Re: [SPAM] Database performance is worse with (DIRECT IO)O_DIRECT

[Ryusuke Konishi <ryusuke-sG5X7nlA6pw@public.gmane.org>]

Hi!
On Wed, 5 Aug 2009 22:59:43 +0900, "Yongkun Wang" wrote:
> hi, guys
> 
> I have been using this awesome file system for a while.
>
> I built a database server on it. The transaction throughput on
> NILFS2 is about several times faster than that on EXT2.
>
> However, when I set the flush method to DIRECT IO (O_DIRECT), the
> database performance on NILFS2 is even worse than that on EXT2 with
> DIRECT IO.

Interesting results.

Nilfs2 is actually not yet supporting O_DIRECT write because the core
routine of direct IO is not available for copy-on-write filesystems.
It is fallen back to a buffered write for nilfs.

O_DIRECT read is OK, but I took O_DIRECT write is implied in your
case.  Do you use postgresql ?

> My conjecture is that probably the log-structured file system need a
> lot of buffer to apply changes and group the writes.

Likely reason.  You can see what's happing on the partition by using
``dumpseg'' command; it will show summary of logs for a specified
segment.  ( You can track recently written segments with ``lssu''
command )

If your conjecture is right, you will see many files and meta data
files with inode number 4, 5, and 3, are written very frequently.

> Is it also possible that the implementation of NILFS2 on DIRECT IO
> is not very efficient?
>
> Before I read the code, could you give me some hints why the
> performance is not good with DIRECT IO?

Yes, as I mentioned above.

I think we can add native support of O_DIRECT to nilfs, but I suspect
there is a influential difference in how to flush out data between the
native direct-io and the fall-backed buffered write & sync; we may be
able to improve performance without the native o-direct support.


> Or could you please introduce the implementation relevant to the
> DIRECT IO?

Roughly speaking, I think the current nilfs implementation of O_DIRECT
leads to the same action as O_DSYNC.  To be exact, we need to analyze
the disk with the dumpseg or trace internal functions because the
behavior depends on the environment.

Cheers,
Ryusuke Konishi