Device Mapper as a cache for SAN?

Device Mapper as a cache for SAN?

[Eric Van Hensbergen]

It seems like device-mapper would allow a fairly straightforward
implementation of a local-disk cache for a SAN (iSCSI, AOE, etc.).  
The dm-raid1 and dm-mpath modules seem to have elements of overlap,
but its not clear if there's a straightforward way to implement a
local-disk block cache using existing modules, or whether a new one is
needed.  Has anyone looked at this before?

          -eric

Re: Device Mapper as a cache for SAN?

[Ed Wilts]

On Mon, Apr 10, 2006 at 12:58:36PM -0500, Eric Van Hensbergen wrote:
> It seems like device-mapper would allow a fairly straightforward
> implementation of a local-disk cache for a SAN (iSCSI, AOE, etc.).  
> The dm-raid1 and dm-mpath modules seem to have elements of overlap,
> but its not clear if there's a straightforward way to implement a
> local-disk block cache using existing modules, or whether a new one is
> needed.  Has anyone looked at this before?

The obvious question is why?  What would this provide that the kernel
and file systems don't already provide?

        .../Ed

-- 
Ed Wilts, RHCE
Mounds View, MN, USA
mailto:ewilts@ewilts.org
Member #1, Red Hat Community Ambassador Program

Re: Device Mapper as a cache for SAN?

[Eric Van Hensbergen]

On 4/10/06, Ed Wilts <ewilts@ewilts.org> wrote:
>
> The obvious question is why?  What would this provide that the kernel
> and file systems don't already provide?
>

Well, kernel provides in-memory page cache, not local disk page cache.
 There is some ongoing remote file system local disk cache work being
done, but it applies primarily to AFS and NFS - not block-based
transports such as iSCSI and AOE.  The various cluster file systems
may provide something more akin to this, but again, this doesn't
really cover SAN technologies -- not in any sort of a generic way. 
Let me know if I'm missing something.

        -eric

Re: Device Mapper as a cache for SAN?

[Ming Zhang]

On Mon, 2006-04-10 at 14:46 -0500, Eric Van Hensbergen wrote:
> On 4/10/06, Ed Wilts <ewilts@ewilts.org> wrote:
> >
> > The obvious question is why?  What would this provide that the kernel
> > and file systems don't already provide?
> >
> 
> Well, kernel provides in-memory page cache, not local disk page cache.
>  There is some ongoing remote file system local disk cache work being
> done, but it applies primarily to AFS and NFS - not block-based
> transports such as iSCSI and AOE.  The various cluster file systems
> may provide something more akin to this, but again, this doesn't
> really cover SAN technologies -- not in any sort of a generic way. 
> Let me know if I'm missing something.

still, what is the main benefit of this cache? what u mean "local disk
page cache"? use "local disk" as page cache or page cache for "local
disk" or something else?

ming


> 
>         -eric
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Device Mapper as a cache for SAN?

[Eric Van Hensbergen]

On 4/10/06, Ming Zhang <mingz@ele.uri.edu> wrote:
> On Mon, 2006-04-10 at 14:46 -0500, Eric Van Hensbergen wrote:
> > On 4/10/06, Ed Wilts <ewilts@ewilts.org> wrote:
> > >
> > > The obvious question is why?  What would this provide that the kernel
> > > and file systems don't already provide?
> > >
> >
> > Well, kernel provides in-memory page cache, not local disk page cache.
> >  There is some ongoing remote file system local disk cache work being
> > done, but it applies primarily to AFS and NFS - not block-based
> > transports such as iSCSI and AOE.  The various cluster file systems
> > may provide something more akin to this, but again, this doesn't
> > really cover SAN technologies -- not in any sort of a generic way.
> > Let me know if I'm missing something.
>
> still, what is the main benefit of this cache? what u mean "local disk
> page cache"? use "local disk" as page cache or page cache for "local
> disk" or something else?
>

Okay, I suppose I wasn't exactly clear.  What I want is something that
will let me use a local disk (or local disks) as a cache for a storage
area network volume (or volumes).  There are a variety of scenarios -
my primary interest is in using it for large clusters with shared
read-only volumes or with exclusive-access read/write volumes.   So,
local disk page cache is probably not the right terminology -- it is
simply a local block cache for a remote block device.

       -eric

Re: Device Mapper as a cache for SAN?

[Ming Zhang]

On Mon, 2006-04-10 at 19:28 -0500, Eric Van Hensbergen wrote:
> On 4/10/06, Ming Zhang <mingz@ele.uri.edu> wrote:
> > On Mon, 2006-04-10 at 14:46 -0500, Eric Van Hensbergen wrote:
> > > On 4/10/06, Ed Wilts <ewilts@ewilts.org> wrote:
> > > >
> > > > The obvious question is why?  What would this provide that the kernel
> > > > and file systems don't already provide?
> > > >
> > >
> > > Well, kernel provides in-memory page cache, not local disk page cache.
> > >  There is some ongoing remote file system local disk cache work being
> > > done, but it applies primarily to AFS and NFS - not block-based
> > > transports such as iSCSI and AOE.  The various cluster file systems
> > > may provide something more akin to this, but again, this doesn't
> > > really cover SAN technologies -- not in any sort of a generic way.
> > > Let me know if I'm missing something.
> >
> > still, what is the main benefit of this cache? what u mean "local disk
> > page cache"? use "local disk" as page cache or page cache for "local
> > disk" or something else?
> >
> 
> Okay, I suppose I wasn't exactly clear.  What I want is something that
> will let me use a local disk (or local disks) as a cache for a storage
> area network volume (or volumes).  There are a variety of scenarios -

so a disk as cache for another (remote) disk(s).

> my primary interest is in using it for large clusters with shared
> read-only volumes or with exclusive-access read/write volumes.   So,
> local disk page cache is probably not the right terminology -- it is
> simply a local block cache for a remote block device.

yes, "page cache" has its meaning in linux.

as i know, no stable implementation for this.

> 
>        -eric

Re: Device Mapper as a cache for SAN?

[Eric Van Hensbergen]

On 4/10/06, Ming Zhang <mingz@ele.uri.edu> wrote:
>
> yes, "page cache" has its meaning in linux.
>

Sorry - my bad, I should have known better.

>
> as i know, no stable implementation for this.
>

Okay - I'll take a shot at it - should be a good intro to
device-mapper as a framework.  If anyone else is going down this path,
please shoot me an email so I don't end up duplicating effort.

       -eric

Re: Device Mapper as a cache for SAN?

[Ming Zhang]

On Mon, 2006-04-10 at 19:47 -0500, Eric Van Hensbergen wrote:
> On 4/10/06, Ming Zhang <mingz@ele.uri.edu> wrote:
> >
> > yes, "page cache" has its meaning in linux.
> >
> 
> Sorry - my bad, I should have known better.
> 
> >
> > as i know, no stable implementation for this.
> >
> 
> Okay - I'll take a shot at it - should be a good intro to
> device-mapper as a framework.  If anyone else is going down this path,
> please shoot me an email so I don't end up duplicating effort.

good luck. a disk cache for remote storage system is not always useful,
considering u already have GB RAM here at file system level. of course,
depends on your workload pattern and remote access cost, u might get
some marginal benefits.


> 
>        -eric

Re: Device Mapper as a cache for SAN?

[Jure Pečar]

On Mon, 10 Apr 2006 17:25:25 -0400
Ming Zhang <mingz@ele.uri.edu> wrote:

> still, what is the main benefit of this cache? what u mean "local disk
> page cache"? use "local disk" as page cache or page cache for "local
> disk" or something else?

I recently did some tests with ext3 data journaling and discovered that
large journals eat almost as much system memory as its size, which is a
bit crazy. Some might argue that configuring 4gb journal is crazy too,
but there are scenarios where this might be beneficial (think large
mail servers with gazillion small writes, which journal turns into one
big sequential write). So if something like external journal can be
implemented in dm, fs-agnostic and not as memory hungry, I'd be very
interested.

-- 

Jure Pečar
http://jure.pecar.org

Re: Device Mapper as a cache for SAN?

[Ming Zhang]

On Tue, 2006-04-11 at 10:55 +0200, Jure Pečar wrote:
> On Mon, 10 Apr 2006 17:25:25 -0400
> Ming Zhang <mingz@ele.uri.edu> wrote:
> 
> > still, what is the main benefit of this cache? what u mean "local disk
> > page cache"? use "local disk" as page cache or page cache for "local
> > disk" or something else?
> 
> I recently did some tests with ext3 data journaling and discovered that
> large journals eat almost as much system memory as its size, which is a
> bit crazy. Some might argue that configuring 4gb journal is crazy too,
> but there are scenarios where this might be beneficial (think large
> mail servers with gazillion small writes, which journal turns into one
> big sequential write). So if something like external journal can be
> implemented in dm, fs-agnostic and not as memory hungry, I'd be very
> interested.

even u journal is a big sequential write, u data write are not. so the
combined performance will not have big difference. 

u mean u journal contain everything? including log and data? you might
want to ask ext3 list and ask them why take so many memory with a 4GB
journal. Also i think journal will write to disk very fast, so after
written to disk, the ram can be reclaimed.

also i do not see how a DM disk cache can help you on this.

ming

Re: Device Mapper as a cache for SAN?

[Artur Bergman]

[-- Attachment #1.1: Type: text/plain, Size: 677 bytes --]


On 10 Apr 2006, at 12:46, Eric Van Hensbergen wrote:

>  The various cluster file systems
> may provide something more akin to this, but again, this doesn't
> really cover SAN technologies -- not in any sort of a generic way.
> Let me know if I'm missing something.

I think everyone asking is missing something.

kernels cache, applications that use raw disk cache, SAN devices  
cache, I fail to see what you really are proposing and therefore I  
fail to see what value you are proposing?

Caching a SAN to local disk? usually my SANs are quite a bit faster  
than local disk, caching to memory? what does that buy you that the  
kernel/application caching gets you?

Artur

[-- Attachment #1.2: Type: text/html, Size: 1617 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]