GFS2/OCFS2 scalability

GFS2/OCFS2 scalability

[Kirill Kuvaldin]

Hi,

I'm evaluating different cluster file systems that can work with large
clustered environment, e.g. hundreds of nodes connected to a SAN over
FC.


 So far I looked at OCFS2 and GFS2, they both worked nearly the same
in terms of performance, but since I ran my tests in a local limited
environment with 4 nodes in a cluster, the results can't be
extrapolated to 100 nodes.

Is there any reading on comparison of OCFS2/GFS2 scalability? And
could these FS scale up to environment of that size (100-1000 nodes,
10-100Tb storage) or should I look to proprietary solutions like IBM
GPFS, HP Polyserve, etc... ?

Thank you,

Kirill

Re: GFS2/OCFS2 scalability

[Andreas Dilger]

On Feb 20, 2009  20:23 +0300, Kirill Kuvaldin wrote:
> I'm evaluating different cluster file systems that can work with large
> clustered environment, e.g. hundreds of nodes connected to a SAN over
> FC.
> 
> 
>  So far I looked at OCFS2 and GFS2, they both worked nearly the same
> in terms of performance, but since I ran my tests in a local limited
> environment with 4 nodes in a cluster, the results can't be
> extrapolated to 100 nodes.
> 
> Is there any reading on comparison of OCFS2/GFS2 scalability? And
> could these FS scale up to environment of that size (100-1000 nodes,
> 10-100Tb storage) or should I look to proprietary solutions like IBM
> GPFS, HP Polyserve, etc... ?

I'm fairly biased, but I think you should look at Lustre - it is currently
running on 15 of the 20 fastest systems in the world, and 40% of the top 200.
No problems with scalability at all - the ORNL Jaguar system has 37000+
nodes, 10PB of storage, and has exceeded 150GB/s peak read/write speed.

Lustre is GPL and freely available: http://www.lustre.org/

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: GFS2/OCFS2 scalability

[Sunil Mushran]

Andreas Dilger wrote:
> On Feb 20, 2009  20:23 +0300, Kirill Kuvaldin wrote:
>> I'm evaluating different cluster file systems that can work with large
>> clustered environment, e.g. hundreds of nodes connected to a SAN over
>> FC.
>>
>>  So far I looked at OCFS2 and GFS2, they both worked nearly the same
>> in terms of performance, but since I ran my tests in a local limited
>> environment with 4 nodes in a cluster, the results can't be
>> extrapolated to 100 nodes.
>>
>> Is there any reading on comparison of OCFS2/GFS2 scalability? And
>> could these FS scale up to environment of that size (100-1000 nodes,
>> 10-100Tb storage) or should I look to proprietary solutions like IBM
>> GPFS, HP Polyserve, etc... ?
> I'm fairly biased, but I think you should look at Lustre - it is currently
> running on 15 of the 20 fastest systems in the world, and 40% of the top 200.
> No problems with scalability at all - the ORNL Jaguar system has 37000+
> nodes, 10PB of storage, and has exceeded 150GB/s peak read/write speed.
>
> Lustre is GPL and freely available: http://www.lustre.org/

I am fairly biased too. However, I don't believe any shared disk cluster 
file
system can scale upto 1000 nodes. This includes ocfs2 and gfs2. I don't know
much about gpfs to comment.

Lustre is parallel distributed and not shared disk. It can scale to 1000s of
nodes but it requires more hardware commitment. Different data and meta data
servers, for starters. But if you want 1000 nodes, then it makes sense to
invest in that kind of hardware.

But I am a bit puzzled by your statement that ocfs2 and gfs2 perform
similarly. That can't be. Everyone knows ocfs2 beats gfs2 hands down. ;)

Sunil

Re: GFS2/OCFS2 scalability

[Kirill Kuvaldin]

On Sat, Feb 21, 2009 at 4:59 AM, Sunil Mushran <sunil.mushran@oracle.com> wrote:
>
> I am fairly biased too. However, I don't believe any shared disk cluster
> file
> system can scale upto 1000 nodes. This includes ocfs2 and gfs2. I don't know
> much about gpfs to comment.

What are the practical/theoretical limits for number of nodes for
shared disk file systems like ocfs2/gfs2?

>
> Lustre is parallel distributed and not shared disk. It can scale to 1000s of
> nodes but it requires more hardware commitment. Different data and meta data
> servers, for starters. But if you want 1000 nodes, then it makes sense to
> invest in that kind of hardware.
>
> But I am a bit puzzled by your statement that ocfs2 and gfs2 perform
> similarly. That can't be. Everyone knows ocfs2 beats gfs2 hands down. ;)

I did testing in a limited setup - a cluster of 4 xen domU of 128MB
RAM with a shared block device mapped to a local LVM volume. My
results could be wrong anyway ;)

Are there any publicly available benchmarks?

Thanks,
Kirill

Re: GFS2/OCFS2 scalability

[Sunil Mushran]

Kirill Kuvaldin wrote:
> What are the practical/theoretical limits for number of nodes for
> shared disk file systems like ocfs2/gfs2?

Theoretical limit is around 254 or so. Practical limit depends on
the hardware. Meaning, you cannot just add nodes. You have to
ensure the interconnect and the storage can handle the nodes. Also,
the more nodes you have, the more cpu/ram each node will have to dedicate
to the clustering overhead. Meaning, at some cluster size, dual core
nodes may not give you the best bang.

So with a GigE interconnect, 2/4G Fiber storage and dual core nodes,
32 to 64 nodes _may_ be the upper limit to the cluster size. A lot
depends on the workload... meaning hard numbers are not possible.

As far as clusters in use go, I have heard of 32 node ocfs2 clusters. Few.
More common is <= 20. 16 is quite common. mkfs.ocfs2 now defaults to 8...
meaning 8 is the very common.

> I did testing in a limited setup - a cluster of 4 xen domU of 128MB
> RAM with a shared block device mapped to a local LVM volume. My
> results could be wrong anyway ;)

You cannot really prototype a 100-1000 node cluster with 4 128MB
xen domUs.