Re: Incremental Backup of repositories using Git

["Michal Suchánek" <msuchanek@suse.de>]

On Thu, May 08, 2025 at 03:47:31PM -0400, Jeff King wrote:
> On Thu, May 08, 2025 at 08:47:47PM +0200, Michal Suchánek wrote:
> 
> > If you have one of those filesystems that support deduplication on
> > filesystem level you could make each snapshot as a full repository with
> > all objects unpacked, and the filesystem would deduplicate the objects
> > for you.
> > 
> > The downside is that you have no way to do multiple full backups this
> > way, and you would have to use something else for that (such as those
> > bundles, or plain archiving the repository as files in a tar archive or
> > such.
> 
> This is tempting, but I suspect that storing the objects unpacked will
> become unfeasibly large, because you are missing out on delta
> compression in the packfiles. You can compare the on-disk and
> uncompressed sizes of objects in a repo like this:
> 
>   git cat-file --batch-all-objects --unordered \
>                --batch-check='%(objectsize:disk) %(objectsize)' |
>   perl -alne '
>     $disk += $F[0];
>     $true += $F[1];
>     END {
>       print "$true / $disk = ", int($true / $disk);
>     }
>   '
> 
> It's not entirely fair because the "true" size is missing out on zlib
> compression that loose objects would get. But that's at best going to be
> about 4:1 (and in practice worse, since trees are full of sha1 hashes
> that don't compress very well).
> 
> In my copy of linux.git, that yields ~135G versus ~2.4G, for a factor of
> 56. Even if we grant 4:1 compression from zlib, that's still inflating
> your on-disk repository by a factor of 14.

So with this estimate you recoup that size inflation after 14
incremental backups.

Since no other working incremantal backup strategy was proposed so far
this is the best one ;-)

Thanks

Michal