Could device-mapper offer multi-tier storage?

Could device-mapper offer multi-tier storage?

[Ty! Boyack]

Hi all,

I'm seeing a good bit of interest in multi-tier storage in high end 
systems, and was wondering if this functionality could be offered by 
device-mapper.

By multi-tier, I'm speaking of systems where various speed/quality 
storage units are used in an analogous way to the way cache is used for 
memory:  The most recently/often used data blocks are placed on the 
fastest storage device, then over time gets flushed out to slower 
systems.  It seems the trend is to use a smaller array of 15k rpm SAS 
drives as the first tier, then flush it out to much larger arrays of 
7200 rpm SATA drives as the second tier.

While this clearly has benefits in large systems, I would think the same 
concept could be brought in-box as well, by using a fast 15k or solid 
state drive as the first tier, and then pushing out to 7200 rpm SATA as 
a second tier. 

My interest is more in the large system side, but if this functionality 
was available and easy to use, I could see it getting wide adoption in 
smaller systems too -- just as RAID was once limited to big systems, 
this could be a cool technology to make available to all.

So the questions:

1)  Is device-mapper the right layer to look at applying this?

2)  If so, any ideas on how hard it would be to implement?

3)  If device-mapper is not the right place, any idea where this 
functionality should go, or if it is already out there in another form?

Thanks -- I would love to hear any thoughts or questions on this.

-Ty!




-- 
-===========================-
  Ty! Boyack
  NREL Unix Network Manager
  ty@nrel.colostate.edu
  (970) 491-1186
-===========================-

Re: Could device-mapper offer multi-tier storage?

[Jonathan Brassow]

I wonder how much Heinz's replicator code could work with this...

Heinz et al have a "remote replicator" target that 'logs' requests to  
a local device and then pushes them to remote devices.  I could  
imagine the log device being the fast device, and the remote devices  
being the slow devices.

So, I'm not sure how much overlap there is here, but it's something to  
think about.

  brassow

On Jan 14, 2009, at 1:43 PM, Ty! Boyack wrote:

> Hi all,
>
> I'm seeing a good bit of interest in multi-tier storage in high end  
> systems, and was wondering if this functionality could be offered by  
> device-mapper.
>
> By multi-tier, I'm speaking of systems where various speed/quality  
> storage units are used in an analogous way to the way cache is used  
> for memory:  The most recently/often used data blocks are placed on  
> the fastest storage device, then over time gets flushed out to  
> slower systems.  It seems the trend is to use a smaller array of 15k  
> rpm SAS drives as the first tier, then flush it out to much larger  
> arrays of 7200 rpm SATA drives as the second tier.
>
> While this clearly has benefits in large systems, I would think the  
> same concept could be brought in-box as well, by using a fast 15k or  
> solid state drive as the first tier, and then pushing out to 7200  
> rpm SATA as a second tier.
> My interest is more in the large system side, but if this  
> functionality was available and easy to use, I could see it getting  
> wide adoption in smaller systems too -- just as RAID was once  
> limited to big systems, this could be a cool technology to make  
> available to all.
>
> So the questions:
>
> 1)  Is device-mapper the right layer to look at applying this?
>
> 2)  If so, any ideas on how hard it would be to implement?
>
> 3)  If device-mapper is not the right place, any idea where this  
> functionality should go, or if it is already out there in another  
> form?
>
> Thanks -- I would love to hear any thoughts or questions on this.
>
> -Ty!
>
>
>
>
> -- 
> -===========================-
> Ty! Boyack
> NREL Unix Network Manager
> ty@nrel.colostate.edu
> (970) 491-1186
> -===========================-
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Could device-mapper offer multi-tier storage?

[Hannes Reinecke]

Hi Jon et al,

Jonathan Brassow wrote:
> I wonder how much Heinz's replicator code could work with this...
> 
> Heinz et al have a "remote replicator" target that 'logs' requests to a 
> local device and then pushes them to remote devices.  I could imagine 
> the log device being the fast device, and the remote devices being the 
> slow devices.
> 
> So, I'm not sure how much overlap there is here, but it's something to 
> think about.
> 
Well, maybe. Not so sure if device-mapper is the correct place to implement
something like this.

What you get if you use a device-mapper device as the 'middle' tier in
this scenario is that this device will contain only random blocks,
ie those actually accessed/modified/whatever.
Only the underlying device will contain the full information about
all blocks, and as such only the underlying device can be mounted
and provides a meaningful filesystem.
Any middle tier devices will contain a somewhat random update of
several blocks, which cannot be interpreted at all.

This makes error checking _really_ hard, as the middle tier devices
can't be consistency checked at all; we have to hope the blocks on
there make sense to the underlying device.

No, this scenario could be more efficiently handled on the filesystem
level, as we then have an implicit consistency check as you're restricted
to move _files_ to a different storage; but that's okay as the filesystems
in on each device themselves will be consistent.

Would be a grand application for union mount if and when it comes around.
And probably btrfs already has it built-in, I wouldn't wonder.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: Could device-mapper offer multi-tier storage?

[malahal]

Hannes Reinecke [hare@suse.de] wrote:
> Well, maybe. Not so sure if device-mapper is the correct place to implement
> something like this.
>
> What you get if you use a device-mapper device as the 'middle' tier in
> this scenario is that this device will contain only random blocks,
> ie those actually accessed/modified/whatever.
> Only the underlying device will contain the full information about
> all blocks, and as such only the underlying device can be mounted
> and provides a meaningful filesystem.
> Any middle tier devices will contain a somewhat random update of
> several blocks, which cannot be interpreted at all.
>
> This makes error checking _really_ hard, as the middle tier devices
> can't be consistency checked at all; we have to hope the blocks on
> there make sense to the underlying device.

Are you proposing a "dm-cache" target here? That may work as HSM
(hierarchical Storage management) if dm-cache target can accept tier'ed
devices. But it has to keep track of access times on the blocks to move
them among tier'ed devices. Can be made to work, but I think it is
efficient to do it in the file system as you indicated.

>
> No, this scenario could be more efficiently handled on the filesystem
> level, as we then have an implicit consistency check as you're restricted
> to move _files_ to a different storage; but that's okay as the filesystems
> in on each device themselves will be consistent.
>
> Would be a grand application for union mount if and when it comes around.
> And probably btrfs already has it built-in, I wouldn't wonder.

I am not sure if "union mount" is really needed as the file system
driver can implicitly take care of HSM with a single mount.

Re: Could device-mapper offer multi-tier storage?

[Christoph Hellwig]

On Wed, Jan 14, 2009 at 12:43:31PM -0700, Ty! Boyack wrote:
> 1)  Is device-mapper the right layer to look at applying this?
>
> 2)  If so, any ideas on how hard it would be to implement?
>
> 3)  If device-mapper is not the right place, any idea where this  
> functionality should go, or if it is already out there in another form?

The right way to do it is in the filesystem.  There's already something
like this in XFS although you need the out of tree version.  dmapi
is designed to work with really slow media like tapes and can migrate
data fully out to it.

There's something else under development which allows to assign
different regions of the filesystem to files and hiearchies based
on various criteria.  This is called dynamic allocation policies and
will initially use dm for creating the underlying device hiearchy,
although we'd eventually want the raid/volume manager algorithms tied
more tightly directly into the fs control.  The exact way to do that
is still a little bit under discussion.