[linux-lvm] Distributed LVM/filesystem/storage

[linux-lvm] Distributed LVM/filesystem/storage

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

Hi!

I'm just thinking about using my friend's overly empty harddisks for a
common large filesystem by merging them all together into a single,
large storage pool accessible by everybody.

However I still fail to find a good approach. One initial idea was to
export all physical volumes (partitions or full disks) vie NBD, join
them all on a single machine into one VG/LV, create a filesystem and
re-export that via NFS/SMB/...  However, all data would need to go
through the net twice. While speed isn't the most important thing here
(but storage is!), that's way suboptimal.

I also thought about something like tahoe (www.allmydata.org), or
other P2P based approaches like Freenet, but I'm not sure if that's a
good start.

It would be nice to see if anybody of you did the same before (merging
the free space from a lot computers into one commonly used large
filesystem), if it was successful and what techniques
(LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get there,
and how well that worked out in the end.

Thanks, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:               http://www.eyrie.org/~eagle/faqs/questions.html
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: [linux-lvm] Distributed LVM/filesystem/storage

[Gerrard Geldenhuis]

Maybe have a look at GFS.

Regards

> -----Original Message-----
> From: linux-lvm-bounces@redhat.com
[mailto:linux-lvm-bounces@redhat.com]
> On Behalf Of Jan-Benedict Glaw
> Sent: 30 May 2008 00:12
> To: linux-lvm@redhat.com
> Subject: [linux-lvm] Distributed LVM/filesystem/storage
> 
> Hi!
> 
> I'm just thinking about using my friend's overly empty harddisks for a
> common large filesystem by merging them all together into a single,
> large storage pool accessible by everybody.
> 
> However I still fail to find a good approach. One initial idea was to
> export all physical volumes (partitions or full disks) vie NBD, join
> them all on a single machine into one VG/LV, create a filesystem and
> re-export that via NFS/SMB/...  However, all data would need to go
> through the net twice. While speed isn't the most important thing here
> (but storage is!), that's way suboptimal.
> 
> I also thought about something like tahoe (www.allmydata.org), or
> other P2P based approaches like Freenet, but I'm not sure if that's a
> good start.
> 
> It would be nice to see if anybody of you did the same before (merging
> the free space from a lot computers into one commonly used large
> filesystem), if it was successful and what techniques
> (LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get
there,
> and how well that worked out in the end.
> 
> Thanks, JBG
> 
> --
>       Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-
> 7608481
> Signature of:
> http://www.eyrie.org/~eagle/faqs/questions.html
> the second  :

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

On Fri, 2008-05-30 09:03:35 +0100, Gerrard Geldenhuis <Gerrard.Geldenhuis@datacash.com> wrote:
> On Behalf Of Jan-Benedict Glaw
> > I'm just thinking about using my friend's overly empty harddisks for a
> > common large filesystem by merging them all together into a single,
> > large storage pool accessible by everybody.
[...]
> > It would be nice to see if anybody of you did the same before (merging
> > the free space from a lot computers into one commonly used large
> > filesystem), if it was successful and what techniques
> > (LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get there,
> > and how well that worked out in the end.
>
> Maybe have a look at GFS.

GFS (or GFS2 fwiw) imposes a single, shared storage as its backend. At
least I get that from reading the documentation. This would result in
merging all the single disks via NBD/LVM to one machine first and
export that merged volume back via NBD/iSCSI to the nodes. In case the
actual data is local to a client, it would still be first send to the
central machine (running LVM) and loaded back from there. Not as
distributed as I hoped, or are there other configuration possibilities
to not go that route?

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:         Alles wird gut! ...und heute wirds schon ein bißchen besser.
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Stuart D. Gathman]

On Sat, 31 May 2008, Jan-Benedict Glaw wrote:

> GFS (or GFS2 fwiw) imposes a single, shared storage as its backend. At
> least I get that from reading the documentation. This would result in
> merging all the single disks via NBD/LVM to one machine first and
> export that merged volume back via NBD/iSCSI to the nodes. In case the
> actual data is local to a client, it would still be first send to the
> central machine (running LVM) and loaded back from there. Not as
> distributed as I hoped, or are there other configuration possibilities
> to not go that route?

Any possible scheme combining disks over the network is going to
send the data over the network twice.  The solution is simple -
put the low level disk traffic on its own network.  Consider it
a long distance SCSI bus.  If you think about it, normal NFS disk traffic
goes over a "network" twice also.  Once over the ethernet, and again
over the IDE/SCSI/SATA "network".

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Wendy Cheng]

Jan-Benedict Glaw wrote:
> On Fri, 2008-05-30 09:03:35 +0100, Gerrard Geldenhuis <Gerrard.Geldenhuis@datacash.com> wrote:
>   
>> On Behalf Of Jan-Benedict Glaw
>>     
>>> I'm just thinking about using my friend's overly empty harddisks for a
>>> common large filesystem by merging them all together into a single,
>>> large storage pool accessible by everybody.
>>>       
> [...]
>   
>>> It would be nice to see if anybody of you did the same before (merging
>>> the free space from a lot computers into one commonly used large
>>> filesystem), if it was successful and what techniques
>>> (LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get there,
>>> and how well that worked out in the end.
>>>       
>> Maybe have a look at GFS.
>>     
>
> GFS (or GFS2 fwiw) imposes a single, shared storage as its backend. At
> least I get that from reading the documentation. This would result in
> merging all the single disks via NBD/LVM to one machine first and
> export that merged volume back via NBD/iSCSI to the nodes. In case the
> actual data is local to a client, it would still be first send to the
> central machine (running LVM) and loaded back from there. Not as
> distributed as I hoped, or are there other configuration possibilities
> to not go that route?
>   

GFS is certainly developed and well tuned in a SAN environment where the 
shared storage(s) and cluster nodes reside on the very same fibre 
channel switch network. However, with its symmetric architecture, 
nothing can prevent it running on top of a group of iscsi disks (with 
GFS node as initiator), as long as each node can see and access these 
disks. It doesn't care where the iscsi targets live, nor how many there 
are. Of course, whether it can perform well in this environment is 
another story. In short, the notion that GFS requires all disks to be 
merged into one machine first and then export the merged volume back to 
the GFS node is *not* correct.

I actually have a 4-nodes cluster in my house. Two nodes running Linux 
iscsi initiators that have a 2-node GFS cluster setup. Another two nodes 
running a special version of  free-BSD as iscsi targets, each directly 
exports their local disks to the GFS nodes. I have not put too much IO 
loads on the GFS nodes though (since the cluster is mostly used to study 
storage block allocation issues - not for real data and/or application).

cc linxu-cluster

-- Wendy

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 2183 bytes --]

On Sat, 2008-05-31 23:12:21 -0500, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
> Jan-Benedict Glaw wrote:
> > On Fri, 2008-05-30 09:03:35 +0100, Gerrard Geldenhuis <Gerrard.Geldenhuis@datacash.com> wrote:
> > > On Behalf Of Jan-Benedict Glaw
> > > > I'm just thinking about using my friend's overly empty harddisks for a
> > > > common large filesystem by merging them all together into a single,
> > > > large storage pool accessible by everybody.
> >
> > [...]
> >  
> > > > It would be nice to see if anybody of you did the same before (merging
> > > > the free space from a lot computers into one commonly used large
> > > > filesystem), if it was successful and what techniques
> > > > (LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get there,
> > > > and how well that worked out in the end.
> > >
> > > Maybe have a look at GFS.
> >
> > GFS (or GFS2 fwiw) imposes a single, shared storage as its backend. At
> > least I get that from reading the documentation. This would result in
> > merging all the single disks via NBD/LVM to one machine first and
> > export that merged volume back via NBD/iSCSI to the nodes. In case the
> > actual data is local to a client, it would still be first send to the
> > central machine (running LVM) and loaded back from there. Not as
> > distributed as I hoped, or are there other configuration possibilities
> > to not go that route?
> 
>                         However, with its symmetric architecture, 
> nothing can prevent it running on top of a group of iscsi disks (with 
> GFS node as initiator), as long as each node can see and access these 
> disks. It doesn't care where the iscsi targets live, nor how many there 
> are.

So I'd configure each machine's empty disk/partition as an iSCSI
target and let them show up an every "client" machine and run that
setup. How good will GFS deal with temporary (or total) outage of
single targets? Eg. 24h disconnects with ADSL connectivity etc.?

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:                http://catb.org/~esr/faqs/smart-questions.html
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Wendy Cheng]

Jan-Benedict Glaw wrote:
> On Sat, 2008-05-31 23:12:21 -0500, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
>   
>> Jan-Benedict Glaw wrote:
>>     
>>> On Fri, 2008-05-30 09:03:35 +0100, Gerrard Geldenhuis <Gerrard.Geldenhuis@datacash.com> wrote:
>>>       
>>>> On Behalf Of Jan-Benedict Glaw
>>>>         
>>>>> I'm just thinking about using my friend's overly empty harddisks for a
>>>>> common large filesystem by merging them all together into a single,
>>>>> large storage pool accessible by everybody.
>>>>>           
>>> [...]
>>>  
>>>       
>>>>> It would be nice to see if anybody of you did the same before (merging
>>>>> the free space from a lot computers into one commonly used large
>>>>> filesystem), if it was successful and what techniques
>>>>> (LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get there,
>>>>> and how well that worked out in the end.
>>>>>           
>>>> Maybe have a look at GFS.
>>>>         
>>> GFS (or GFS2 fwiw) imposes a single, shared storage as its backend. At
>>> least I get that from reading the documentation. This would result in
>>> merging all the single disks via NBD/LVM to one machine first and
>>> export that merged volume back via NBD/iSCSI to the nodes. In case the
>>> actual data is local to a client, it would still be first send to the
>>> central machine (running LVM) and loaded back from there. Not as
>>> distributed as I hoped, or are there other configuration possibilities
>>> to not go that route?
>>>       
>>                         However, with its symmetric architecture, 
>> nothing can prevent it running on top of a group of iscsi disks (with 
>> GFS node as initiator), as long as each node can see and access these 
>> disks. It doesn't care where the iscsi targets live, nor how many there 
>> are.
>>     
>
> So I'd configure each machine's empty disk/partition as an iSCSI
> target and let them show up an every "client" machine and run that
> setup. How good will GFS deal with temporary (or total) outage of
> single targets? Eg. 24h disconnects with ADSL connectivity etc.?
>
>   
High availability will not work well in this particular setup - it is 
more about data and storage sharing between GFS nodes.

Note that GFS normally runs on top of CLVM (clustered lvm, in case you 
don't know about it). You might want to check current (Linux) CLVM raid 
level support to see whether it fits your needs. 
 

-- Wendy

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]

On Sun, 2008-06-01 08:50:26 -0500, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
> Jan-Benedict Glaw wrote:
> > On Sat, 2008-05-31 23:12:21 -0500, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
> > > Jan-Benedict Glaw wrote:
> > > > On Fri, 2008-05-30 09:03:35 +0100, Gerrard Geldenhuis <Gerrard.Geldenhuis@datacash.com> wrote:
> > > > > On Behalf Of Jan-Benedict Glaw
> > > > > > I'm just thinking about using my friend's overly empty harddisks for a
> > > > > > common large filesystem by merging them all together into a single,
> > > > > > large storage pool accessible by everybody.
> > > > [...]
> > > > > > It would be nice to see if anybody of you did the same before (merging
> > > > > > the free space from a lot computers into one commonly used large
> > > > > > filesystem), if it was successful and what techniques
> > > > > > (LVM/NBD/DM/MD/iSCSI/Tahoe/Freenet/Other P2P/...) you used to get 
> > > > > > there, and how well that worked out in the end.
> > > > >
> > > > > Maybe have a look at GFS.
> >
> > So I'd configure each machine's empty disk/partition as an iSCSI
> > target and let them show up an every "client" machine and run that
> > setup. How good will GFS deal with temporary (or total) outage of
> > single targets? Eg. 24h disconnects with ADSL connectivity etc.?
> 
> High availability will not work well in this particular setup - it is 
> more about data and storage sharing between GFS nodes.
> 
> Note that GFS normally runs on top of CLVM (clustered lvm, in case you 
> don't know about it). You might want to check current (Linux) CLVM raid 
> level support to see whether it fits your needs. 

As I expect nodes to come and leave the filesystem and that these
nodes all provide different amounts of disk capacity, reshaping the
RAID stuff several times a week (or even per day) won't be that much
fun at all...

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of:                 Friends are relatives you make for yourself.
the second  :

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Distributed LVM/filesystem/storage

[David Brown]

On Sat, May 31, 2008 at 08:42:09PM -0400, Stuart D. Gathman wrote:

>Any possible scheme combining disks over the network is going to
>send the data over the network twice.  The solution is simple -
>put the low level disk traffic on its own network.  Consider it
>a long distance SCSI bus.  If you think about it, normal NFS disk traffic
>goes over a "network" twice also.  Once over the ethernet, and again
>over the IDE/SCSI/SATA "network".

It seems that in a proper cluster filesystem and a proper cluster volume
manager, each node would hold the volume meta data and only need to access
the drives containing the data it needs.  The data shouldn't have to move
over the network twice, only directly between the node holding the disk and
the node using the data.  That's actually kind of the whole point of a
cluster filesystem.

The hard part is the locking so that these nodes can all update the
filesystem without hopelessly scrambling it.

   <http://en.wikipedia.org/wiki/Shared_disk_file_system>

David

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Hannes Dorbath]

Jan-Benedict Glaw wrote:
> However I still fail to find a good approach. One initial idea was to
> export all physical volumes (partitions or full disks) vie NBD, join
> them all on a single machine into one VG/LV, create a filesystem and
> re-export that via NFS/SMB/...  However, all data would need to go
> through the net twice. While speed isn't the most important thing here
> (but storage is!), that's way suboptimal.

GlusterFS with Unify translator is exactly what you are asking for.

http://www.gluster.org/docs/index.php/GlusterFS


-- 
Best regards,
Hannes Dorbath

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

On Sun, 2008-06-01 23:09:01 +0200, Hannes Dorbath <light@theendofthetunnel.de> wrote:
> Jan-Benedict Glaw wrote:
> > However I still fail to find a good approach. One initial idea was to
> > export all physical volumes (partitions or full disks) vie NBD, join
> > them all on a single machine into one VG/LV, create a filesystem and
> > re-export that via NFS/SMB/...  However, all data would need to go
> > through the net twice. While speed isn't the most important thing here
> > (but storage is!), that's way suboptimal.
> 
> GlusterFS with Unify translator is exactly what you are asking for.
> 
> http://www.gluster.org/docs/index.php/GlusterFS

Already had a look at that, too. "Unify" seems to not directly
implement any measure of redundancy. So if one Brick goes down,
everything that was there is gone. Maybe its possible to play with the
`mirror' translator as well, but on the first look, it seems to be
non-trivial to accomodate for a steady loss and addition of bricks.

MfG, JBG

-- 
      Jan-Benedict Glaw      jbglaw@lug-owl.de              +49-172-7608481
Signature of: 23:53 <@jbglaw> So, ich kletter' jetzt mal ins Bett.
the second  : 23:57 <@jever2> .oO( kletter ..., hat er noch Gitter vorm Bett, wie früher meine Kinder?)
              00:00 <@jbglaw> jever2: *patsch*
              00:01 <@jever2> *aua*, wofür, Gedanken sind frei!
              00:02 <@jbglaw> Nee, freie Gedanken, die sind seit 1984 doch aus!
              00:03 <@jever2> 1984? ich bin erst seit 1985 verheiratet!

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Distributed LVM/filesystem/storage

[Greg Freemyer]

2008/6/1 Jan-Benedict Glaw <jbglaw@lug-owl.de>:
> On Sun, 2008-06-01 23:09:01 +0200, Hannes Dorbath <light@theendofthetunnel.de> wrote:
>> Jan-Benedict Glaw wrote:
>> > However I still fail to find a good approach. One initial idea was to
>> > export all physical volumes (partitions or full disks) vie NBD, join
>> > them all on a single machine into one VG/LV, create a filesystem and
>> > re-export that via NFS/SMB/...  However, all data would need to go
>> > through the net twice. While speed isn't the most important thing here
>> > (but storage is!), that's way suboptimal.
>>
>> GlusterFS with Unify translator is exactly what you are asking for.
>>
>> http://www.gluster.org/docs/index.php/GlusterFS
>
> Already had a look at that, too. "Unify" seems to not directly
> implement any measure of redundancy. So if one Brick goes down,
> everything that was there is gone. Maybe its possible to play with the
> `mirror' translator as well, but on the first look, it seems to be
> non-trivial to accomodate for a steady loss and addition of bricks.
>
> MfG, JBG

I haven't seen Lustre mentioned in this thread.  I think there is
kernel support, etc.  May be worth checking out. (I have not used it.)

Quoting wikipedia http://en.wikipedia.org/wiki/Lustre_(file_system)
===
High availability

Lustre file system high availability features include a robust
failover and recovery mechanism, making server failures and reboots
transparent. Version interoperability between successive minor
versions of the Lustre software enables a server to be upgraded by
taking it offline (or failing it over to a standby server), performing
the upgrade, and restarting it, while all active jobs continue to run,
merely experiencing a delay.

Lustre MDSs are configured as an active/passive pair, while OSSs are
typically deployed in an active/active configuration that provides
redundancy without extra overhead. Often the standby MDS is the active
MDS for another Lustre file system, so no nodes are idle in the
cluster.
===

Greg
-- 
Greg Freemyer
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com