Re: Content based storage

[Fabio <tux@ilbello.com>]

Some years ago I was searching for that kind of functionality and found 
an experimental ext3 patch to allow the so-called COW-links: 
http://lwn.net/Articles/76616/

There was a discussion later on LWN http://lwn.net/Articles/77972/
an approach like COW-links would break POSIX standards.

I am not very technical and don't know if it's feasible in btrfs.
I think most likely you'll have to run an userspace tool to find and 
merge identical files based on checksums (which already sounds good to me).
The only thing we can ask the developers at the moment is if something 
like that would be possible without changes to the on-disk format.


PS. Another great scenario is shared hosting web/file servers: ten of 
thousand website with mostly the same tiny PHP Joomla files.
If you can get the benefits of: compression + "content based"/cowlinks + 
FS Cache... That would really make Btrfs FLY on Hard Disk and make SSD 
devices possible for storage (because of the space efficiency).

-- 
Fabio


David Brown ha scritto:
> Hi,
>
> I was wondering if there has been any thought or progress in 
> content-based storage for btrfs beyond the suggestion in the "Project 
> ideas" wiki page?
>
> The basic idea, as I understand it, is that a longer data extent 
> checksum is used (long enough to make collisions unrealistic), and 
> merge data extents with the same checksums.  The result is that "cp 
> foo bar" will have pretty much the same effect as "cp --reflink foo 
> bar" - the two copies will share COW data extents - as long as they 
> remain the same, they will share the disk space.  But you can still 
> access each file independently, unlike with a traditional hard link.
>
> I can see at least three cases where this could be a big win - I'm 
> sure there are more.
>
> Developers often have multiple copies of source code trees as 
> branches, snapshots, etc.  For larger projects (I have multiple 
> "buildroot" trees for one project) this can take a lot of space.  
> Content-based storage would give the space efficiency of hard links 
> with the independence of straight copies.  Using "cp --reflink" would 
> help for the initial snapshot or branch, of course, but it could not 
> help after the copy.
>
> On servers using lightweight virtual servers such as OpenVZ, you have 
> multiple "root" file systems each with their own copy of "/usr", etc. 
> With OpenVZ, all the virtual roots are part of the host's file system 
> (i.e., not hidden within virtual disks), so content-based storage 
> could merge these, making them very much more efficient.  Because each 
> of these virtual roots can be updated independently, it is not 
> possible to use "cp --reflink" to keep them merged.
>
> For backup systems, you will often have multiple copies of the same 
> files.  A common scheme is to use rsync and "cp -al" to make 
> hard-linked (and therefore space-efficient) snapshots of the trees.  
> But sometimes these things get out of synchronisation - perhaps your 
> remote rsync dies halfway, and you end up with multiple independent 
> copies of the same files.  Content-based storage can then re-merge 
> these files.
>
>
> I would imagine that content-based storage will sometimes be a 
> performance win, sometimes a loss.  It would be a win when merging 
> results in better use of the file system cache - OpenVZ virtual 
> serving would be an example where you would be using multiple copies 
> of the same file at the same time.  For other uses, such as backups, 
> there would be no performance gain since you seldom (hopefully!) read 
> the backup files.  But in that situation, speed is not a major issue.
>
>
> mvh.,
>
> David
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html