COW files

COW files

[Gregory Maxwell]

Any thought to making a file plugin that creates copy on write files?
The operation would be something like a hardlink which is invisible to
the user and broken as soon as either file is modified.

Files could be COWed by a flag on the cp command (or really, perhaps
that should be the default behavior) or with a utility (perhaps run as
a periodic script to locate duplicates and COW them. This would
greatly speed up the process of copying files.

The behavior on break could be to duplicate the whole cowed file on
the first right, or allow a COWed file to have alternate choices for
blocks. Files would remain cowed until decowed, which would likely be
bad for performance (due to fragmentation of alternate versions
causing gaps in sequential scans), so the repacker could be taught to
decow files that have too many alternate blocks.

Re: COW files

[David Masover]

Gregory Maxwell wrote:

> Files could be COWed by a flag on the cp command (or really, perhaps
> that should be the default behavior) or with a utility (perhaps run as
> a periodic script to locate duplicates and COW them. This would
> greatly speed up the process of copying files.

Should be on the cp command, but you have to change the cp source for
that, and everything implementing "copy", including desktop managers.
There's currently no "copy" system call.  So it's more than just a plugin.

> The behavior on break could be to duplicate the whole cowed file on
> the first right, or allow a COWed file to have alternate choices for
> blocks. Files would remain cowed until decowed, which would likely be
> bad for performance (due to fragmentation of alternate versions
> causing gaps in sequential scans), so the repacker could be taught to
> decow files that have too many alternate blocks.

Not sure about this last one.  The repacker should increase the amount
of free space, not decrease it.  Especially, imagine a very large COWed
file, maybe a quarter the size of the FS, with 15 or so copies.  Yes,
it'll be fragmented, but the alternative is an FS full several times over.

Re: COW files

[Lexington Luthor]

David Masover wrote:
> Gregory Maxwell wrote:
> 
>> Files could be COWed by a flag on the cp command (or really, perhaps
>> that should be the default behavior) or with a utility (perhaps run as
>> a periodic script to locate duplicates and COW them. This would
>> greatly speed up the process of copying files.
> 
> Should be on the cp command, but you have to change the cp source for
> that, and everything implementing "copy", including desktop managers.
> There's currently no "copy" system call.  So it's more than just a plugin.
> 

I would love something like this for entire directory trees. Perhaps 
with a reiser4 specific tool that does a COW cp -al.

At the moment, I have to cp -al and use a userspace solution 
(LD_PRELOADed library which traps the open call) to get working COW on 
reiser4.

I am not concerned too much with space usage, but speed is very 
important. This is usually not a problem with reiser4 because it so 
rarely hits the disk with COW trees.

- LL

Re: COW files

[David Masover]

Lexington Luthor wrote:
> David Masover wrote:
> 
>> Gregory Maxwell wrote:
>>
>>> Files could be COWed by a flag on the cp command (or really, perhaps
>>> that should be the default behavior) or with a utility (perhaps run as
>>> a periodic script to locate duplicates and COW them. This would
>>> greatly speed up the process of copying files.
>>
>>
>> Should be on the cp command, but you have to change the cp source for
>> that, and everything implementing "copy", including desktop managers.
>> There's currently no "copy" system call.  So it's more than just a
>> plugin.
>>
> 
> I would love something like this for entire directory trees. Perhaps
> with a reiser4 specific tool that does a COW cp -al.

Ok, are you talking about making all the files COW, or the directory
structure itself somehow COW?

Because if someone patched "cp" to do COW files, I'm sure it would also
work for "cp -a", which would be about as fast as "cp -al".

I'm not sure how it'd work to make entire directories COW -- that is,
"cp -a" is as fast as "ln" of a single file.


And before anyone from LKML gets any funny ideas, there's no reason why
COW should be Reiser4 specific.  We've been due a "copy" system call for
at least a decade.  Why not just add that, with the realization that
POSIX would be brain-dead not to add support for it later, then make it
COW for FS-es that support it, and a simple copy for FS-es that don't.

With any luck, not a line of any non-COW FS need be changed to support
the copy system call, unless they want to add COW support as well.

And if you think it's bad copying a couple of gigs around on a local
hard disk, wishing it could copy-on-write, try an NFS mount over a
(very) long-distance VPN.  Why should cp send a couple of gigs across a
network and back, when the machine running the "cp" command never needs
to see it?

Re: COW files

[Lexington Luthor]

David Masover wrote:
> Ok, are you talking about making all the files COW, or the directory
> structure itself somehow COW?
> 
> Because if someone patched "cp" to do COW files, I'm sure it would also
> work for "cp -a", which would be about as fast as "cp -al".
> 
> I'm not sure how it'd work to make entire directories COW -- that is,
> "cp -a" is as fast as "ln" of a single file.
> 
> 
> And before anyone from LKML gets any funny ideas, there's no reason why
> COW should be Reiser4 specific.  We've been due a "copy" system call for
> at least a decade.  Why not just add that, with the realization that
> POSIX would be brain-dead not to add support for it later, then make it
> COW for FS-es that support it, and a simple copy for FS-es that don't.
> 
> With any luck, not a line of any non-COW FS need be changed to support
> the copy system call, unless they want to add COW support as well.
> 
> And if you think it's bad copying a couple of gigs around on a local
> hard disk, wishing it could copy-on-write, try an NFS mount over a
> (very) long-distance VPN.  Why should cp send a couple of gigs across a
> network and back, when the machine running the "cp" command never needs
> to see it?
> 

I would not mind this being reiser4 specific, I don't use the archaic 
filesystems much, they are far too slow compared to reiser4.

As for the exact semantics of COW, whether it applies to file or to 
entire directories, I don't really care as long as it is fast. At the 
moment, the userspace library makes things very fast, but very 
inconvenient since I need to spawn a new shell in the hard-linked tree, 
and while screen + zsh and its history sharing makes that bearable, its 
still a PITA.

LL

Re: COW files

[Hans Reiser]

Gregory Maxwell wrote:

>Any thought to making a file plugin that creates copy on write files?
>The operation would be something like a hardlink which is invisible to
>the user and broken as soon as either file is modified.
>
>Files could be COWed by a flag on the cp command (or really, perhaps
>that should be the default behavior) or with a utility (perhaps run as
>a periodic script to locate duplicates and COW them. This would
>greatly speed up the process of copying files.
>
>The behavior on break could be to duplicate the whole cowed file on
>the first right, or allow a COWed file to have alternate choices for
>blocks. Files would remain cowed until decowed, which would likely be
>bad for performance (due to fragmentation of alternate versions
>causing gaps in sequential scans), so the repacker could be taught to
>decow files that have too many alternate blocks.
>
>
>  
>
we are open to patches for this, and nobody is writing one....;-)