[LSF/MM TOPIC] [ATTEND] HSM, DMAPI, and external media filesystems

[LSF/MM TOPIC] [ATTEND] HSM, DMAPI, and external media filesystems

[Ben Myers]

I would like to discuss how to get DMAPI accepted into the mainline
kernel.  DMAPI is or was supported by a number of filesystems which are
included in linux (including XFS) and also is supported by a number of
proprietary filesystems.  However, the DMAPI implementation has not yet
been accepted into the main line.

- What are the barriers to inclusion and how can they be resolved?

- What FOSS HSM projects are available to use the interface?
	* xfstests/dmapi/sample_hsm
	* openhsm (opensms)
	* tinyhsm (unfinished, unreleased)

- Who are the interested parties (if any), and how could we split up the
  work?

In the end I'd like to come up with a plan that seems reasonable toward
getting DMAPI included mainline.  I know that Alex Elder and others have
done work in this area and I would like to see it bear fruit.

---

Secondly, I'd enjoy a discussion of an old idea from back in the day
when I worked as a sysadmin and became a storage software hobbyist.
Having recently lost a great deal of data due to disk failure and failed
backups, I struck upon the idea of turning the backup paradigm on it's
head:

I'd rather have the authoritative copy of my data on an external medium
and only cache it on spinning rust.  The filesystem would log every
transaction (WORM style) to removable media and the set of discs or
tapes would be replayable onto new hard disks when I had a failure.
Checkpoints would ensure that the number of discs/tapes necessary to
replay the filesystem remained manageable, and the write performance
problem would be mitigated by maintaining a temporary log on spinning
disk to spool up enough data to keep the tape from having to rewind.
One could choose to recycle media from before a checkpoint, or save it
off for posterity.

My choice of medium at the time was cd-rom since I couldn't afford a
tape drive.  I went about trying to implement such a thing using DMAPI,
which isn't suitable (you can't set an event mask on a file quickly
enough to log the first writes).  Then I tried a stacking filesystem,
fuse, got a job, and lost interest.

This is almost certainly not a new idea.  If there is already something
like this implemented, I'd like to hear about it.  If there isn't, maybe
we could come up with a rough design on a cocktail napkin and pique
enough interest to make a start.

Regards,
Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [LSF/MM TOPIC] [ATTEND] HSM, DMAPI, and external media filesystems

[Greg Freemyer]

On Thu, Jan 19, 2012 at 2:39 PM, Ben Myers <bpm@sgi.com> wrote:
> - What FOSS HSM projects are available to use the interface?
>        * xfstests/dmapi/sample_hsm
>        * openhsm (opensms)
>        * tinyhsm (unfinished, unreleased)

* ohsm (unfinished, unreleased) (I think)
http://code.google.com/p/fscops/
Presented at OLS 2009

Greg

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [LSF/MM TOPIC] [ATTEND] HSM, DMAPI, and external media filesystems

[Manish Katiyar]

> * ohsm (unfinished, unreleased) (I think)
> http://code.google.com/p/fscops/
> Presented at OLS 2009

Latest one was at http://sourceforge.net/projects/ohsm/

-- 
Thanks -
Manish

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [LSF/MM TOPIC] [ATTEND] HSM, DMAPI, and external media filesystems

[Christoph Hellwig]

On Thu, Jan 19, 2012 at 01:39:47PM -0600, Ben Myers wrote:
> I would like to discuss how to get DMAPI accepted into the mainline
> kernel.  DMAPI is or was supported by a number of filesystems which are
> included in linux (including XFS) and also is supported by a number of
> proprietary filesystems.  However, the DMAPI implementation has not yet
> been accepted into the main line.
> 
> - What are the barriers to inclusion and how can they be resolved?

The grotty DMAPI interface.  That includes both the high-level ioctl
implementations as well as lower level details.

In the end most of it boils down to three things:

 - a notification API telling the daemon about changes, and allowing
   it to block them.  That's in some ways fairly similar to fanotify,
   which should be reused.  Preferably including the interface, but at
   the very least the underlying implementation.
 - a handle based I/O method.  We now have open by handle at the VFS,
   so it might just need a few additions like the invisible I/O flag.
 - ways to make extents offline in the fs, and set various HSM specific
   flags.  This is the actual filesystem-specific piece of code, but
   we'll need to think about an interface to query and set these bits.
   filesystem-specific piece of code.

> 
> - What FOSS HSM projects are available to use the interface?
> 	* xfstests/dmapi/sample_hsm

That actually seems to be the only one.

> 	* openhsm (opensms)

While that one has HSM in the name it actually seems to work quite
differently from the traditional partially offline HSM model, nevermind
that it seems abandoned.

> 	* tinyhsm (unfinished, unreleased)

pointers?

> In the end I'd like to come up with a plan that seems reasonable toward
> getting DMAPI included mainline.  I know that Alex Elder and others have
> done work in this area and I would like to see it bear fruit.

I'm not really sure discussing this grand highlevel design without
backing code really fits for LSF.  Might be a fine topic for a BOF
if enough interested people are around.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs