1 Gb Ethernet based HPC storage deployment plan

1 Gb Ethernet based HPC storage deployment plan

[Lee Eric]

Hi all,

I got 20 nodes cluster(1 Gb Ethernet connected) and gonna build a
small storage system with it. Because of the low budget I'm thinking
to use 1 Gb Ethernet as physical layer for the storage part. Here's
what I planed.

There 8 SATA hard drivers so I wanna use AoE or iSCSI to export the
block devices to the master node over Ethernet. Then the master node
will use NFS to export block devices to compute nodes. The upper layer
file system I will use XFS.

I know it would be better if I can use Lustre but my interconnection
is a little slow. I suspect if it is feasible if using such parallel
file system.

Does anyone has good idea on this deployment?

Thanks very much.

Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 1 Gb Ethernet based HPC storage deployment plan

[Emmanuel Florac]

Le Wed, 20 Jul 2011 22:03:33 +0800 vous écriviez:

> I know it would be better if I can use Lustre but my interconnection
> is a little slow. I suspect if it is feasible if using such parallel
> file system.
> 
> Does anyone has good idea on this deployment?
> 

For this kind of setup, true cluster filesystems like Lustre,
PVFS2/OrangeFS, Gluster, Ceph... would be much better. Striping 20
iSCSI volumes across would be awfully dangerous.

I'd go with OrangeFS (pvfs.org) because I'm pretty happy with it so far
(using XFS as the underlying local filesystem). It's precisely made to
agreggate computing clusters storage.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 1 Gb Ethernet based HPC storage deployment plan

[Stan Hoeppner]

On 7/23/2011 4:30 AM, Emmanuel Florac wrote:
> Le Wed, 20 Jul 2011 22:03:33 +0800 vous écriviez:
> 
>> I know it would be better if I can use Lustre but my interconnection
>> is a little slow. I suspect if it is feasible if using such parallel
>> file system.
>>
>> Does anyone has good idea on this deployment?
>>
> 
> For this kind of setup, true cluster filesystems like Lustre,
> PVFS2/OrangeFS, Gluster, Ceph... would be much better. Striping 20
> iSCSI volumes across would be awfully dangerous.
> 
> I'd go with OrangeFS (pvfs.org) because I'm pretty happy with it so far
> (using XFS as the underlying local filesystem). It's precisely made to
> agreggate computing clusters storage.

Typically one starts looking at hardware solutions after identifying the
needs of the target application/workload.

Is the proposed storage cluster system simply a proof of concept
testbed, or will it actually be tasked with real work?  If the latter
I'd rethink your iSCSI export to NFS server idea.  You mentioned only 8
disks.  Just drop them directly into the NFS host and avoid many
potential headaches down the road.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 1 Gb Ethernet based HPC storage deployment plan

[Lee Eric]

Thanks mates. So the typical storage solution for the small size
cluster may use IP SAN as I know before. Yes, I can export the data by
using NFS directly without iSCSI/AoE but is there any good point to
use XFS? I just know XFS is better for parallelized read/write
operations in local disks.

By the way, is there any good advantage to use XFS as the underlying
local filesystem for cluster/distributed/parallel filesystem?

Thanks very much.

Eric

On Sun, Jul 24, 2011 at 2:30 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 7/23/2011 4:30 AM, Emmanuel Florac wrote:
>> Le Wed, 20 Jul 2011 22:03:33 +0800 vous écriviez:
>>
>>> I know it would be better if I can use Lustre but my interconnection
>>> is a little slow. I suspect if it is feasible if using such parallel
>>> file system.
>>>
>>> Does anyone has good idea on this deployment?
>>>
>>
>> For this kind of setup, true cluster filesystems like Lustre,
>> PVFS2/OrangeFS, Gluster, Ceph... would be much better. Striping 20
>> iSCSI volumes across would be awfully dangerous.
>>
>> I'd go with OrangeFS (pvfs.org) because I'm pretty happy with it so far
>> (using XFS as the underlying local filesystem). It's precisely made to
>> agreggate computing clusters storage.
>
> Typically one starts looking at hardware solutions after identifying the
> needs of the target application/workload.
>
> Is the proposed storage cluster system simply a proof of concept
> testbed, or will it actually be tasked with real work?  If the latter
> I'd rethink your iSCSI export to NFS server idea.  You mentioned only 8
> disks.  Just drop them directly into the NFS host and avoid many
> potential headaches down the road.
>
> --
> Stan
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 1 Gb Ethernet based HPC storage deployment plan

[Stan Hoeppner]

On 7/25/2011 6:52 AM, Lee Eric wrote:
> Thanks mates. So the typical storage solution for the small size
> cluster may use IP SAN as I know before. Yes, I can export the data by
> using NFS directly without iSCSI/AoE but is there any good point to
> use XFS? I just know XFS is better for parallelized read/write
> operations in local disks.
> 
> By the way, is there any good advantage to use XFS as the underlying
> local filesystem for cluster/distributed/parallel filesystem?

Narrow down your candidate list of distributed filesystems and read the
documentation for each of them.  I'd guess that each one of them has a
recommendation of some sort for the local storage node filesystem and
the reasoning behind the recommendation.  Given the manner in which most
of them derive their parallel performance, the local filesystem is
likely not critical.

You mentioned an IP SAN.  Have you looked at GFS2 and OCFS?  You haven't
mentioned a workload.  We could better serve you if you described the
workload.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 1 Gb Ethernet based HPC storage deployment plan

[Lee Eric]

Thanks Stan. Well, because the size of cluster is not very big so
here's what I considered on its storage deployment. The users home
directory and data will be stored in OpenAFS. And also as I mentioned
in the first thread, 8 SATA disks will be used by using IP SAN to
share with compute nodes.

Some file systems I considered but not sure. Lustre, I know it has
good performance but I just use GigaEther network environment in this
cluster hence I am not pretty sure the performance would be good
seemed only high speed storage connected network can get good result.
GFS2, yeah, I ever heard about some institutes use such distributed
file systems for cluster computing field but still no evidence for its
scalability and performance. So my simple way just to use XFS as
underlayer and export it by using NFS.

For the real workload, I run bioinformatics software actually. They
may write many large or small files as parallel computing to the
storage.

Thanks.

Eric

On Mon, Jul 25, 2011 at 9:58 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 7/25/2011 6:52 AM, Lee Eric wrote:
>> Thanks mates. So the typical storage solution for the small size
>> cluster may use IP SAN as I know before. Yes, I can export the data by
>> using NFS directly without iSCSI/AoE but is there any good point to
>> use XFS? I just know XFS is better for parallelized read/write
>> operations in local disks.
>>
>> By the way, is there any good advantage to use XFS as the underlying
>> local filesystem for cluster/distributed/parallel filesystem?
>
> Narrow down your candidate list of distributed filesystems and read the
> documentation for each of them.  I'd guess that each one of them has a
> recommendation of some sort for the local storage node filesystem and
> the reasoning behind the recommendation.  Given the manner in which most
> of them derive their parallel performance, the local filesystem is
> likely not critical.
>
> You mentioned an IP SAN.  Have you looked at GFS2 and OCFS?  You haven't
> mentioned a workload.  We could better serve you if you described the
> workload.
>
> --
> Stan
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 1 Gb Ethernet based HPC storage deployment plan

[Emmanuel Florac]

Le Mon, 25 Jul 2011 19:52:52 +0800 vous écriviez:

> By the way, is there any good advantage to use XFS as the underlying
> local filesystem for cluster/distributed/parallel filesystem?

It depends upon the cluster FS. PVFS prefers XFS. Other may work
differently.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs