vcs for hefty video and graphics files

vcs for hefty video and graphics files

[Harry Putnam]

I hope not to appear to be sneaking in a war of systems, and may rate
a bit a high in windbag factor since I've asked a very similar
question here but quite a while ago

I'd like to briefly describe my projected usage of a versioning system
and see what people here think in terms what system best suits to that
usage.

I'm a light weight semi-professional videographer but I have enough
going on to feel a need to keep track of and be able to rollback
versions of a project as it is being worked on

It would involve on any one projects something like 15 to 60 GB of
stuff to keep up with.  Large numbers of images and a dozen or 2 dozen
video files.  All in some stake of compression depending on the codex.

I'd like to keep a few version of say an Adobe Premier project with
all the attendant files that play a role in it.

Ditto for sets of Adobe After effects and attendant files.  And many
many images and compilation in various states of cutting or whatever
editing.

And then the whole project including the various Premier and After
effects sets and piles of captured video tape.  All in DV-avi format.

So versions inside versions and then repeated for different projects.

Each of which may have more than 1 version.

But cutting to the chase... assorted video and graphics files
amounting to between 15 - 60 GB in any one project

Each project would only run a month or 2 months at the most and then
all but the final delivered version would be deleted.  That version
might be keep for a yr or so.

I have some experience with cvs over quite a few yrs but still only a
homeboy user and not very skilled or knowledgeable about cvs and even
less with mercurial or git, where I do have some tiny bit of
experience too.

Which of the main contenders:  cvs subversion mercurial git bizarre 
Maybe  a few more I don't know about, would be the best candidate for
the usage and user described

Re: vcs for hefty video and graphics files

[Philippe Lhoste]

On 22/11/2010 19:09, Harry Putnam wrote:
> Which of the main contenders:  cvs subversion mercurial git bizarre
> Maybe  a few more I don't know about, would be the best candidate for
> the usage and user described

bizarre? Never heard of this VCS before...

 > Each project would only run a month or 2 months at the most and then
 > all but the final delivered version would be deleted.  That version
 > might be keep for a yr or so.

Maybe it is pure heresy, but since all you want is to keep temporarily several very big 
and nearly incompressible files where diffs (or deltas?) are probably not significant, I 
would advance that a VCS won't be very useful here.
Advantages of VCS are (among others):
- make delta of changes to keep as little data as possible
- compress this data (?)
- keep changes indefinitely to be sure to have them when we need them
- share and merge (changes from somebody else, or you elsewhere)
Unless I missed something, these advantages doesn't seem to apply there.

Some game makers keep track of their (large) binary files, along with the rest of the 
project (source code). Rarely in isolation.
Perforce and PlasticSCM both boast superior support of these files, I won't comment on 
these allegations (over other VCS), just having no experience here.

Somehow, in your case, the good old way of keeping copies renamed to keep the version (or 
kept in specific directories) might work for you... Perhaps along with a small text file 
with comments on content of each file.

PS.: I don't see why you included Tomcat list...

-- 
Philippe Lhoste
--  (near) Paris -- France
--  http://Phi.Lho.free.fr
--  --  --  --  --  --  --  --  --  --  --  --  --  --

Re: vcs for hefty video and graphics files

[Harry Putnam]

Philippe Lhoste <PhiLho@GMX.net> writes:

> Somehow, in your case, the good old way of keeping copies renamed to
> keep the version (or kept in specific directories) might work for
> you... Perhaps along with a small text file with comments on content
> of each file.

Thanks for the input... that method has sort of worked so far but
somewhat tedious

> PS.: I don't see why you included Tomcat list...

All I can say about that is that it was NOT my doing.
I only included bazaar cvs mercurial, git and subversion.

Its a mystery to me too.  I hadn't noticed until you mentioned it.
I removed it from this reply.

A copy of my OP is kept on my home machine.  It shows exactly what I
sent out, I'm not sure what it goes thru when it gets to gmane server:

 From: Harry Putnam <reader@newsguy.com>
 Subject: vcs for hefty video and graphics files
 Newsgroups: gmane.comp.version-control.mercurial.general,
 gmane.comp.version-control.bazaar-ng.general,
 gmane.comp.version-control.git,
 gmane.comp.version-control.subversion.user,
 gmane.comp.version-control.cvs.general
 Date: Mon, 22 Nov 2010 12:09:14 -0600
 Message-ID: <877hg55iyd.fsf@newsguy.com>

  

Re: vcs for hefty video and graphics files

[Jonathan Nieder]

Philippe Lhoste wrote:

> Some game makers keep track of their (large) binary files, along
> with the rest of the project (source code). Rarely in isolation.
> Perforce and PlasticSCM both boast superior support of these files,
> I won't comment on these allegations (over other VCS), just having
> no experience here.

One small thing to add: for this use case, you might like git-annex[1].

It is a shame no single mailing list to follow up to was mentioned.
Hopefully the reply will get to Harry, anyway.

Regards,
Jonathan

[1] http://git-annex.branchable.com/

Re: vcs for hefty video and graphics files

[Jakub Narebski]

Jonathan Nieder <jrnieder@gmail.com> writes:
> Philippe Lhoste wrote:
> 
> > Some game makers keep track of their (large) binary files, along
> > with the rest of the project (source code). Rarely in isolation.
> > Perforce and PlasticSCM both boast superior support of these files,
> > I won't comment on these allegations (over other VCS), just having
> > no experience here.
> 
> One small thing to add: for this use case, you might like git-annex[1].
> 
> It is a shame no single mailing list to follow up to was mentioned.
> Hopefully the reply will get to Harry, anyway.
> 
> Regards,
> Jonathan
> 
> [1] http://git-annex.branchable.com/

Jonathan, could you add git-annex to "Interfaces, frontends, and tools"
page on git wiki:
  https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools

Does git-annex use git-replace?

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: vcs for hefty video and graphics files

[Jonathan Nieder]

Jakub Narebski wrote:

> Jonathan, could you add git-annex to "Interfaces, frontends, and tools"
> page on git wiki:
>   https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools

Ok, done.  Please feel free to improve the description.

> Does git-annex use git-replace?

No, just plain symlinks.  See http://git-annex.branchable.com/walkthrough/
to get a feel for it.

Regards,
Jonathan

Re: vcs for hefty video and graphics files

[Stefan Monnier]

> It would involve on any one projects something like 15 to 60 GB of
> stuff to keep up with.  Large numbers of images and a dozen or 2 dozen
> video files.  All in some stake of compression depending on the codex.

FWIW, for a VCS to do a good job on this kind of problem, you'd need
to use a representation that lends itself to it.

I.e. regardless of what you end up doing, I would recommend you contact
the mailing-list of Free Software that can do the kind of video
manipulation you want to do, and tell them that you'd need their tool to
represent a project in such a way that it has a bunch of big-files that
are almost never modified (containing "binary data" such as the source
images and audio recordings, say) along with a few other smaller files
that mostly contain instructions about how to use the big-files to
generate the desired output.

Such a representation should lead to very good support from most VCSs.
E.g. If these small files use a clean text representation, then a VCS
might even be able to do useful merges between different branches of
a project (as long as the branches share the same big-files).


        Stefan