Treat disk space like memory space

Treat disk space like memory space

[Stefan Monnier]

The normal situation on the memory is side is that most of the memory is
in use, but some pages are ready to be discarded, they're just kept
around because we have nothing better to do (yet) with that page.

Is there a tool to do something similar with file systems.  I have a lot
of unimportant files in my file-system, typically browser caches and
things like that, which can be thrown away without any risk of
losing data.  I just keep them around because I have nothing better to
do with my disk space.

So I'd like to be able to say "these areas of my file-system hold data
that you can discard whenever you need space".  So I can freely fill up
my disk with such irrelevant data, safe in the knowledge that if I ever
need this disk space it'll be automatically reclaimed.

[ I realize this is probably better implemented outside of the kernel, but
  it seems like it might be of interest here.  Please redirect me to
  a more appropriate place if you can think of one (other than
  /dev/null that is).  ]


        Stefan

Re: Treat disk space like memory space

[Wes Felter]

Stefan Monnier wrote:

> So I'd like to be able to say "these areas of my file-system hold data
> that you can discard whenever you need space".  So I can freely fill up
> my disk with such irrelevant data, safe in the knowledge that if I ever
> need this disk space it'll be automatically reclaimed.

James Cipar, Mark D. Corner, Emery D. Berger: TFS: A Transparent File 
System for Contributory Storage.

"TFS provides background tasks with large amounts of unreliable 
storage-all of the currently available space-without impacting the 
performance of ordinary file access operations."

http://www.usenix.org/event/fast07/tech/cipar.html
http://prisms.cs.umass.edu/tcsm/

It's intended for P2P but I could imagine using it for caches.

Re: Treat disk space like memory space

[Lee Revell]

On Nov 15, 2007 5:24 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> [ I realize this is probably better implemented outside of the kernel, but
>   it seems like it might be of interest here.  Please redirect me to
>   a more appropriate place if you can think of one (other than
>   /dev/null that is).  ]

It would require some kernel support to reclaim the storage from the
caching application rather than returning -ENOSPC when a "normal" app
needs the storage.  Even if you make the caching app free the space by
itself you need a kernel mechanism to signal it when this happens.

Lee

Re: Treat disk space like memory space

[H. Peter Anvin]

Lee Revell wrote:
> On Nov 15, 2007 5:24 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> [ I realize this is probably better implemented outside of the kernel, but
>>   it seems like it might be of interest here.  Please redirect me to
>>   a more appropriate place if you can think of one (other than
>>   /dev/null that is).  ]
> 
> It would require some kernel support to reclaim the storage from the
> caching application rather than returning -ENOSPC when a "normal" app
> needs the storage.  Even if you make the caching app free the space by
> itself you need a kernel mechanism to signal it when this happens.
> 

In particular, you need a way to hold off the "real" application until 
disk reclaim is done.

If you do it purely in userspace (asynchronously) then it's subject to 
ENOSPC while the reclaimer runs.

This, by the way, has been discussed on and off -- often in the context 
of undelete (which is an identical problem.)  The problem usually is 
that performance of real storage users suffer because of locality 
issues.  However, flash storage doesn't have locality requirements...

	-hpa

Re: Treat disk space like memory space

[Jörn Engel]

On Fri, 16 November 2007 10:30:12 -0800, H. Peter Anvin wrote:
> 
> This, by the way, has been discussed on and off -- often in the context 
> of undelete (which is an identical problem.)  The problem usually is 
> that performance of real storage users suffer because of locality 
> issues.  However, flash storage doesn't have locality requirements...

It does, although significantly less so than disks.  Read latency is
typically between 100x and 1000x less than disk latency.

Another argument against this is that free space directly translates to
speed, both for disks and flash.  Disk filesystems fragment like hell if
the disk is constanly near-full and flash filesystems require a lot more
garbage collection overhead.

Jörn

-- 
To my face you have the audacity to advise me to become a thief - the worst
kind of thief that is conceivable, a thief of spiritual things, a thief of
ideas! It is insufferable, intolerable!
-- M. Binet in Scarabouche

Re: Treat disk space like memory space

[Pavel Machek]

Hi!

> So I'd like to be able to say "these areas of my file-system hold data
> that you can discard whenever you need space".  So I can freely fill up
> my disk with such irrelevant data, safe in the knowledge that if I ever
> need this disk space it'll be automatically reclaimed.
> 
> [ I realize this is probably better implemented outside of the kernel, but
>   it seems like it might be of interest here.  Please redirect me to
>   a more appropriate place if you can think of one (other than
>   /dev/null that is).  ]

Actually, you'll probably need fs hooks for reliable operation. And
yes, someone finally implementing this would be welcome :-).

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html