Performance problems over AoE

Performance problems over AoE

[Delian Krustev]

	Hi All,

Nice to meet you :-)

I'm having performance problems with NILFS2 running on
AoE (Ata over Ethernet) device.

I'm able to get around 5/10 MB sequential write speed for over a device
which is capable of 100MB/s.

I've described the problem and my setup in details on a web page:

http://krustev.net/w/articles/Backup_service_and_software_block_devices_over_the_net/


and will be thankful if you take a look and give some ideas.

And one other question.
Is there a way to disable the auto check point creation ?


Cheers
--
Delian

Re: Performance problems over AoE

[Ryusuke Konishi]

Hi!
On Sat, 19 Dec 2009 02:24:44 +0200, Delian Krustev <krustev-sueGvCUcFGXR7s880joybQ@public.gmane.org> wrote:
> 
> 	Hi All,
> 
> Nice to meet you :-)
> 
> I'm having performance problems with NILFS2 running on
> AoE (Ata over Ethernet) device.
> 
> I'm able to get around 5/10 MB sequential write speed for over a device
> which is capable of 100MB/s.
> 
> I've described the problem and my setup in details on a web page:
> 
> http://krustev.net/w/articles/Backup_service_and_software_block_devices_over_the_net/
> 
> 
> and will be thankful if you take a look and give some ideas.

Thank you for letting us know the interesting measurement result.

Actually we recognize similar issue on high performance drives, but
this looks worse than the cases we know.

In the 2.6.33 merge window, I sent some series of patches upstream to
improve write performance.  And, the effort is still going on.

I would appreciate it if you could try nilfs in kernel 2.6.33-rc1 or
optimization patches that will be appeared ongoingly in this list or
linux-nilfs.

> And one other question.
> Is there a way to disable the auto check point creation ?

Unfortunately no.  The contiguous check point creation is what nilfs
stands for.  But, I think the performance impact of cp creation is not
so high unless you are doing osync writes.
 
> Cheers
> --
> Delian

Cheers,
Ryusuke Konishi

Re: Performance problems over AoE

[Delian Krustev]

	Hi,

Thanks for the answers.

On Sat, 19 Dec 2009 18:43:46 +0900 (JST) Ryusuke Konishi wrote:
> Thank you for letting us know the interesting measurement result.
> 
> Actually we recognize similar issue on high performance drives, but
> this looks worse than the cases we know.

This might be due to the increased latency of the AoE device.

> In the 2.6.33 merge window, I sent some series of patches upstream to
> improve write performance.  And, the effort is still going on.
> 
> I would appreciate it if you could try nilfs in kernel 2.6.33-rc1 or
> optimization patches that will be appeared ongoingly in this list or
> linux-nilfs.

OK, I will try it. Just have to figure out on which hardware since the
setup I've described is on production systems and running early RC kernels
there is not an option.
But I will probably find the metal or will create some VM images.

> Unfortunately no.  The contiguous check point creation is what nilfs
> stands for.  But, I think the performance impact of cp creation is not
> so high unless you are doing osync writes.

Did you read the part for the cleanerd causing high load on the network
interface ?
And the checkpoints being created with no data modifications ?

Do you have a guess for these ?


Cheers
--
Delian

Re: Performance problems over AoE

[Ryusuke Konishi]

Hi,
On Mon, 21 Dec 2009 18:21:39 +0200, Delian Krustev wrote:
> 
> 	Hi,
> 
> Thanks for the answers.
> 
> On Sat, 19 Dec 2009 18:43:46 +0900 (JST) Ryusuke Konishi wrote:
> > Thank you for letting us know the interesting measurement result.
> > 
> > Actually we recognize similar issue on high performance drives, but
> > this looks worse than the cases we know.
> 
> This might be due to the increased latency of the AoE device.

I agree.
 
> > In the 2.6.33 merge window, I sent some series of patches upstream to
> > improve write performance.  And, the effort is still going on.
> > 
> > I would appreciate it if you could try nilfs in kernel 2.6.33-rc1 or
> > optimization patches that will be appeared ongoingly in this list or
> > linux-nilfs.
> 
> OK, I will try it. Just have to figure out on which hardware since the
> setup I've described is on production systems and running early RC kernels
> there is not an option.
> But I will probably find the metal or will create some VM images.

Thank you for your cooperation.

If you prefer customized kernels prior to 2.6.30, I can upload a new
git tree of nilfs2 kmod which backported the latest improvement
patches.

In my debian environment, 2.6.33-rc1 vanilla kernel is, so far,
working nicely.

> > Unfortunately no.  The contiguous check point creation is what nilfs
> > stands for.  But, I think the performance impact of cp creation is not
> > so high unless you are doing osync writes.
> 
> Did you read the part for the cleanerd causing high load on the network
> interface ?

OK, so you mean cp creation by GC.

Yes, it's another issue.  GC of nilfs involves checkpoint creation.
The problem is, the current GC is so stupid and leading to needless
I/O overheads.

You can turn off cleanerd with -i option.

e.g.
 # mount -t nilfs2 -i /dev/xxx /test-dir

And, this is useful when you want to separate the cause of performance
issue.  But anyway, GC should be definitely revised to remedy the
situation.

> And the checkpoints being created with no data modifications ?

Yes, even if you don't modify the filesystem, nilfs internally
requires it for GC.
 
> Do you have a guess for these ?

Well, it looks suffering either or both.  ``mount -i'' would clear up
the matter.

I recognize the current nilfs has both the latency issue and the GC
issue, and I believe both should be improved.

Cheers,
Ryusuke Konishi