Will git have a baseline feature or something alike?

Will git have a baseline feature or something alike?

[eric miao]

All,

I kept a mirror of

http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

by a crontab task fetching the updated commits at midnight everyday.

Yet I found the repository now grows to be 1.2G without checking out
anything. The checked out working tree of this is about 1.5G.

I tried "git prune" and "git repack" but it still remains so large. The
trend of the kernel is still going to be enlarged. Thus I'm thinking
of the possibility of a baseline feature. One can totally forget about
the history before that baseline, and start the development there
after.

E.g.

1. user downloads a released tarball
2. and build a repository
3. and "git fetch" will find the current repository is identical to
a baseline in the remote, and fetches only commits after
that baseline
4. continue the development work

The above steps with current git will generate a totally different
hash value for the files in the downloaded tarball, thus making
it failed to fetch commits thereafter.

I know the history is usually mixed with multiple branches, which
makes this baseline feature a bit difficult to implement. It should
be a nice feature, though.

-- 
Cheers
- eric

Re: Will git have a baseline feature or something alike?

[Sean]

On Fri, 29 Feb 2008 17:23:37 +0800
"eric miao" <eric.y.miao@gmail.com> wrote:


Hi Eric,

> I kept a mirror of
> 
> http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 
> by a crontab task fetching the updated commits at midnight everyday.
> 
> Yet I found the repository now grows to be 1.2G without checking out
> anything. The checked out working tree of this is about 1.5G.

There's something wrong in your setup, the entire kernel history should
take less than 200M.

> I tried "git prune" and "git repack" but it still remains so large. The
> trend of the kernel is still going to be enlarged. Thus I'm thinking
> of the possibility of a baseline feature. One can totally forget about
> the history before that baseline, and start the development there
> after.

Git provides "shallow" clones which essentially give you that ability
today via the --depth option.  While it won't automatically determine
a baseline, it avoids the need for you to download a tarball first.

Sean

Re: Will git have a baseline feature or something alike?

[Karl Hasselström]

On 2008-02-29 04:56:09 -0500, Sean wrote:

> On Fri, 29 Feb 2008 17:23:37 +0800
>
> "eric miao" <eric.y.miao@gmail.com> wrote:
>
> > Yet I found the repository now grows to be 1.2G without checking
> > out anything. The checked out working tree of this is about 1.5G.
>
> There's something wrong in your setup, the entire kernel history
> should take less than 200M.

Try running "git gc".

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: Will git have a baseline feature or something alike?

[Jakub Narebski]

"eric miao" <eric.y.miao@gmail.com> writes:

> I kept a mirror of
> 
> http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 
> by a crontab task fetching the updated commits at midnight everyday.
> 
> Yet I found the repository now grows to be 1.2G without checking out
> anything. The checked out working tree of this is about 1.5G.

Did you (re)packed this repository, running "git gc", or "git repack"?
Currently git either downloads small packs, or loose objects; it needs
to repack to make repository size smaller.

BTW. the largest git repository is 1.6G OpenOffice.org conversion,
with > 2G checkout, and some large binary files under version
control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
So kernel should be quite smaller.
 
> I tried "git prune" and "git repack" but it still remains so large. The
> trend of the kernel is still going to be enlarged. Thus I'm thinking
> of the possibility of a baseline feature. One can totally forget about
> the history before that baseline, and start the development there
> after.

There is so called "shallow clone" feature, which allows to clone only
part of history. Currently it dupports only --depth, i.e. number of
commits from tips; it could I guess support providing tag as
delimiter. (You are welcome to implement it ;-).

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Will git have a baseline feature or something alike?

[Sam Vilain]

Jakub Narebski wrote:
> BTW. the largest git repository is 1.6G OpenOffice.org conversion,
> with > 2G checkout, and some large binary files under version
> control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
> So kernel should be quite smaller.

I have an 8GB git-svn import of the KDE repository :-)

Sam.

Re: Will git have a baseline feature or something alike?

[Jakub Narebski]

On Sat, 1 Mar 2008, Sam Vilain wrote:
> Jakub Narebski wrote:
>
>> BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>> with > 2G checkout, and some large binary files under version
>> control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>> So kernel should be quite smaller.
> 
> I have an 8GB git-svn import of the KDE repository :-)

First, how large full checkout is? And how large Subversion repo?
Second, is this repository tightly packed (large window, big delta
chain)?

And last, KDE repos should most probably be split into submodules.

-- 
Jakub Narebski
Poland

Re: Will git have a baseline feature or something alike?

[Sam Vilain]

Jakub Narebski wrote:
> On Sat, 1 Mar 2008, Sam Vilain wrote:
>> Jakub Narebski wrote:
>>
>>> BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>>> with > 2G checkout, and some large binary files under version
>>> control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>>> So kernel should be quite smaller.
>> I have an 8GB git-svn import of the KDE repository :-)
> 
> First, how large full checkout is? 

Heh, haven't tried tbh.

> And how large Subversion repo?

48GB apparently.  This repository is approx. 90% of the subversion
repository - I didn't prepare it.

> Second, is this repository tightly packed (large window, big delta
> chain)?

Yes, I don't have the details, though.

> And last, KDE repos should most probably be split into submodules.

Mmm.  Everyone always says that; what it really needs I think is someone
to really take the conversion on board and come up with a workable plan
on this front.  I think the counter-argument to this was "but you always
want to have 70% of the repository checked out for development".
Counter-counter argument is "yes but they don't always need to be deep
clones".  Anyway, it's not my baby, just thought I'd let you know about
it :-)

Sam.

Re: Will git have a baseline feature or something alike?

[eric miao]

On Fri, Feb 29, 2008 at 9:21 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> "eric miao" <eric.y.miao@gmail.com> writes:
>
>  > I kept a mirror of
>  >
>  > http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>  >
>  > by a crontab task fetching the updated commits at midnight everyday.
>  >
>  > Yet I found the repository now grows to be 1.2G without checking out
>  > anything. The checked out working tree of this is about 1.5G.
>
>  Did you (re)packed this repository, running "git gc", or "git repack"?
>  Currently git either downloads small packs, or loose objects; it needs
>  to repack to make repository size smaller.
>
>  BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>  with > 2G checkout, and some large binary files under version
>  control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>  So kernel should be quite smaller.
>
>
>  > I tried "git prune" and "git repack" but it still remains so large. The
>  > trend of the kernel is still going to be enlarged. Thus I'm thinking
>  > of the possibility of a baseline feature. One can totally forget about
>  > the history before that baseline, and start the development there
>  > after.
>
>  There is so called "shallow clone" feature, which allows to clone only
>  part of history. Currently it dupports only --depth, i.e. number of
>  commits from tips; it could I guess support providing tag as
>  delimiter. (You are welcome to implement it ;-).
>

I haven't ever used the shallow clone, but it looks still a bit different
from what I thought originally, say, if I download linux-2.6.24.tar.bz2
from kernel.org, that's about 40MB and should be a fair amount.
I then unpack and "git init", I expect it to recognize it's a v2.6.24,
and I can thereafter use "git fetch" to fetch those commits after
v2.6.24 from git.kernel.org. Is this possible?

>  --
>  Jakub Narebski
>  Poland
>  ShadeHawk on #git
>



-- 
Cheers
- eric

Re: Will git have a baseline feature or something alike?

[Nguyen Thai Ngoc Duy]

On Sat, Mar 1, 2008 at 9:10 PM, eric miao <eric.y.miao@gmail.com> wrote:
> On Fri, Feb 29, 2008 at 9:21 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>  > "eric miao" <eric.y.miao@gmail.com> writes:
>  >
>  >  > I kept a mirror of
>  >  >
>  >  > http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>  >  >
>  >  > by a crontab task fetching the updated commits at midnight everyday.
>  >  >
>  >  > Yet I found the repository now grows to be 1.2G without checking out
>  >  > anything. The checked out working tree of this is about 1.5G.
>  >
>  >  Did you (re)packed this repository, running "git gc", or "git repack"?
>  >  Currently git either downloads small packs, or loose objects; it needs
>  >  to repack to make repository size smaller.
>  >
>  >  BTW. the largest git repository is 1.6G OpenOffice.org conversion,
>  >  with > 2G checkout, and some large binary files under version
>  >  control. Mozilla and GCC, other large repos, got under 0.5G IIRC.
>  >  So kernel should be quite smaller.
>  >
>  >
>  >  > I tried "git prune" and "git repack" but it still remains so large. The
>  >  > trend of the kernel is still going to be enlarged. Thus I'm thinking
>  >  > of the possibility of a baseline feature. One can totally forget about
>  >  > the history before that baseline, and start the development there
>  >  > after.
>  >
>  >  There is so called "shallow clone" feature, which allows to clone only
>  >  part of history. Currently it dupports only --depth, i.e. number of
>  >  commits from tips; it could I guess support providing tag as
>  >  delimiter. (You are welcome to implement it ;-).
>  >
>
>  I haven't ever used the shallow clone, but it looks still a bit different
>  from what I thought originally, say, if I download linux-2.6.24.tar.bz2
>  from kernel.org, that's about 40MB and should be a fair amount.
>  I then unpack and "git init", I expect it to recognize it's a v2.6.24,
>  and I can thereafter use "git fetch" to fetch those commits after
>  v2.6.24 from git.kernel.org. Is this possible?

I tried shallow clone (depth 1) with a fairly old linux-2.6 repo and
the pack was 68MB. A bit bigger than 40MB but still acceptable IMO.
The tarball+git-init way, I don't think it work. Maybe kernel.org
could release shallow git bundles in addition to tarballs so  users
like you can download a bundle, make a repo from it and keep up with
"git fetch".

-- 
Duy

Re: Will git have a baseline feature or something alike?

[Jakub Narebski]

On Sat, 1 Mar 2008, eric miao wrote:
> On Fri, Feb 29, 2008 at 9:21 PM, Jakub Narebski <jnareb@gmail.com> wrote:

[could you please remove irrelevant parts of the reply? TIA]

>> There is so called "shallow clone" feature, which allows to clone only
>> part of history. Currently it dupports only --depth, i.e. number of
>> commits from tips; it could I guess support providing tag as
>> delimiter. (You are welcome to implement it ;-).
>>
> 
> I haven't ever used the shallow clone, but it looks still a bit different
> from what I thought originally, say, if I download linux-2.6.24.tar.bz2
> from kernel.org, that's about 40MB and should be a fair amount.
> I then unpack and "git init", I expect it to recognize it's a v2.6.24,
> and I can thereafter use "git fetch" to fetch those commits after
> v2.6.24 from git.kernel.org. Is this possible?

No, this doesn't work and couldn't work. The tarfile contains only
_contents_ of the working directory, and perhaps commit-id, but it
doesn't contain even shred of history. Git has no information of
where this content is in linux kernel git history.

You have to do "git clone --depth=1 <url>"; surrently there is no
way to specify "git clone --from=<tag> <url>". I guess you can try
to get info browsing gitweb, and fudge with grafts.

-- 
Jakub Narebski
Poland

Re: Will git have a baseline feature or something alike?

[Eyvind Bernhardsen]

On 1. mars. 2008, at 16.41, Jakub Narebski wrote:

> On Sat, 1 Mar 2008, eric miao wrote:
>
>> I haven't ever used the shallow clone, but it looks still a bit  
>> different
>> from what I thought originally, say, if I download  
>> linux-2.6.24.tar.bz2
>> from kernel.org, that's about 40MB and should be a fair amount.
>> I then unpack and "git init", I expect it to recognize it's a  
>> v2.6.24,
>> and I can thereafter use "git fetch" to fetch those commits after
>> v2.6.24 from git.kernel.org. Is this possible?
>
> No, this doesn't work and couldn't work. The tarfile contains only
> _contents_ of the working directory, and perhaps commit-id, but it
> doesn't contain even shred of history. Git has no information of
> where this content is in linux kernel git history.

Okay, as a git n00b I'm probably on completely the wrong track, but if  
you made a git repository out of a kernel tarball (cd linux-2.6.24 &&  
git init && git add .) and then did a shallow fetch from kernel.org  
into that repository, wouldn't the blobs you added get reused  
(assuming the tarball you downloaded was fairly recent), thus reducing  
the amount of data fetch has to transfer?

I'm sure you'd end up transferring more data than just a straight  
shallow clone, but it would be cool if that worked, and might even be  
useful.  I'm downloading 2.6.25-rc3 now to try it out :)

Eyvind Bernhardsen

Re: Will git have a baseline feature or something alike?

[Jakub Narebski]

Eyvind Bernhardsen wrote:
>
> Okay, as a git n00b I'm probably on completely the wrong track, but if  
> you made a git repository out of a kernel tarball (cd linux-2.6.24 &&  
> git init && git add .) and then did a shallow fetch from kernel.org  
> into that repository, wouldn't the blobs you added get reused  
> (assuming the tarball you downloaded was fairly recent), thus reducing  
> the amount of data fetch has to transfer?

I think it wouldn't. If I understand it correctly, the fetching engine
deals only with commits. If you have commit, it assumes that you have 
tree, blobs, and ancestors. If you don't have commit, it assumes that 
you don't have tree and blobs.

-- 
Jakub Narebski
Poland

Re: Will git have a baseline feature or something alike?

[David Brown]

On Sat, Mar 01, 2008 at 06:30:13PM +0100, Eyvind Bernhardsen wrote:

> Okay, as a git n00b I'm probably on completely the wrong track, but if you 
> made a git repository out of a kernel tarball (cd linux-2.6.24 && git init 
> && git add .) and then did a shallow fetch from kernel.org into that 
> repository, wouldn't the blobs you added get reused (assuming the tarball 
> you downloaded was fairly recent), thus reducing the amount of data fetch 
> has to transfer?

Git uses the commit history to determine what objects you might already
have.  For normal use cases, this works quite well, however in this
instance it doesn't help at all.  You'll ending up transferring everything,
even for objects you already have.  Git will detect that you already have
the object only after the transfer.

Think about it, though.  In order to do this generally, the client would
have to send the hash of every object it has.  Perhaps this would be a
useful thing to do when git detects that there are no common commits, but
it would only really help the case of pulling from multiple repos that
manage the same files with separate histories.

There are some cases where this happens, such as when changes go through
another revision control system, but probably not normal usage.

David

Re: Will git have a baseline feature or something alike?

[Shawn O. Pearce]

Jakub Narebski <jnareb@gmail.com> wrote:
> Eyvind Bernhardsen wrote:
> >
> > Okay, as a git n00b I'm probably on completely the wrong track, but if  
> > you made a git repository out of a kernel tarball (cd linux-2.6.24 &&  
> > git init && git add .) and then did a shallow fetch from kernel.org  
> > into that repository, wouldn't the blobs you added get reused  
> > (assuming the tarball you downloaded was fairly recent), thus reducing  
> > the amount of data fetch has to transfer?
> 
> I think it wouldn't. If I understand it correctly, the fetching engine
> deals only with commits. If you have commit, it assumes that you have 
> tree, blobs, and ancestors. If you don't have commit, it assumes that 
> you don't have tree and blobs.

Correct.

I was thinking about this just yesterday.  I think that if we
embedded inside of a tarball created by git-archive the raw sources
of all files, plus the commit SHA-1 and the raw body of that commit,
it should be possible to convert that into a shallow clone.

Unfortunately I think it is possible for git-archive to edit a file
in-place during export, e.g. to edit an RPM spec file and insert
the revision.  That would damage the tree, as the blobs would no
longer hash to the same value as they should be in that commit.

-- 
Shawn.

Re: Will git have a baseline feature or something alike?

[Jan Hudec]

On Sun, Mar 02, 2008 at 02:20:35 +1300, Sam Vilain wrote:
> Jakub Narebski wrote:
> > And last, KDE repos should most probably be split into submodules.
> 
> Mmm.  Everyone always says that; what it really needs I think is someone
> to really take the conversion on board and come up with a workable plan
> on this front.  I think the counter-argument to this was "but you always
> want to have 70% of the repository checked out for development".

KDE should not be the case. The source for Debian packages is split in
several independent source packages (which are compilable on their own). You
should rarely need more than 1 or 2 (the one you hack + base) checked out.

> Counter-counter argument is "yes but they don't always need to be deep
> clones".  Anyway, it's not my baby, just thought I'd let you know about
> it :-)

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>

Re: Will git have a baseline feature or something alike?

[Martin Langhoff]

On Sun, Mar 2, 2008 at 2:20 AM, Sam Vilain <sam@vilain.net> wrote:
>  > And last, KDE repos should most probably be split into submodules.
>
>  Mmm.  Everyone always says that; what it really needs I think is someone
>  to really take the conversion on board and come up with a workable plan
>  on this front.  I think the counter-argument to this was "but you always
>  want to have 70% of the repository checked out for development".
>  Counter-counter argument is "yes but they don't always need to be deep
>  clones".  Anyway, it's not my baby, just thought I'd let you know about
>  it :-)

That sounds a lot like the x.org submodule split. AFAIU, it did
involve some relatively minor changes to configure/build scripts to
handle the submodules being built somewhere other than in a subdir of
the main project. Nothing earth shattering, just small changes to how
the build works, and I think after the initial shakeout everyone's
happy and the development pace is quite a bit faster & more
adventurous.

All of this from bits I read in the x.org dev lists, and some
developer blogs - IANAX.OD

cheers,


m