* Re: full kernel history, in patchset format
From: Ingo Molnar @ 2005-04-16 19:41 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>
* Linus Torvalds <torvalds@osdl.org> wrote:
> > the history data starts at 2.4.0 and ends at 2.6.12-rc2. I've included a
> > script that will apply all the patches in order and will create a
> > pristine 2.6.12-rc2 tree.
>
> Hey, that's great. I got the CVS repo too, and I was looking at it,
> but the more I looked at it, the more I felt that the main reason I
> want to import it into git ends up being to validate that my size
> estimates are at all realistic.
>
> I see that Thomas Gleixner seems to have done that already, and come
> to a figure of 3.2GB for the last three years, which I'm very happy
> with, mainly because it seems to match my estimates to a tee. [...]
(yeah, we apparently worked in parallel - i only learned about his
efforts after i sent my mail. He was using BK to extract info, i was
using the CVS tree alone and no BK code whatsoever. (I dont think there
will be any argument about who owns what, but i wanted to be on the safe
side, and i also wanted to see how complete and usable the CVS metadata
is - it's close to perfect i'd say, for the purposes i care about.))
> But I wonder if we actually want to actually populate the whole
> history..
yeah, it definitely feels a bit brave to import 28,000 changesets into a
source-code database project that will be a whopping 2 weeks old in 2
days ;) Even if we felt 100% confident about all the basics (which we do
of course ;), it's just simply too young to tie things down via a 3.2GB
database. It feels much more natural to grow it gradually, 28,000
changesets i'm afraid would just suffocate the 'project growth
dynamics'. Not going too fast is just as important as not going too
slow.
I didnt generate the patchset to get it added into some central
repository right now, i generated it to check that we _do_ have all the
revision history in an easy to understand format which does generate
today's kernel tree, so that we can lean back and worry about the full
database once things get a bit more settled down (in a couple of months
or so). It's also an easy testbed for GIT itself.
but the revision history was one of the main reasons i used BK myself,
so we'll need a merged database eventually. Occasionally i needed to
check who was the one who touched a particular piece of code - was that
fantastic new line of code written by me, or was that buggy piece of
crap written by someone else? ;) Also, looking at a change and then
going to the changeset that did it, and then looking at the full picture
was pretty useful too. So that sort of annotation, and generally
navigating around _quickly_ and looking at the 'flow' of changes going
into a particular file was really useful (for me).
Ingo
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Christopher Li @ 2005-04-16 16:26 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Mike Taht, git
In-Reply-To: <7v8y3i7cn9.fsf@assigned-by-dhcp.cox.net>
We can just have a baseline file contain all the commit objects.
Then have the git "download on demand". The problem with diff
package is that I it is harder to merge with more than one diff.
I bet 90% of the time people sync to the repository head first
want to check out the last bits. And maybe reading some change
log to see what is changed.
So having all the commit object, the user will able to see
what is change and which version he we like to check out.
Then he can issue a command "download me all the objects is needed
for checkout the this commit". Download of demand should be
even better.
Chris
On Sat, Apr 16, 2005 at 12:19:22PM -0700, Junio C Hamano wrote:
> >>>>> "MT" == Mike Taht <mike.taht@timesys.com> writes:
>
> MT> alternatively, "git-archive-torrent" to create a list of files for a
> MT> bittorrent feed....
>
> That is certainly good for establishing the baseline, but you
> still need to leverage the inherent delta-compressibility
> between related blobs/trees by also doing something like what I
> described as "diff package", don't you?
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Thomas Gleixner @ 2005-04-16 20:35 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504161208410.7211@ppc970.osdl.org>
On Sat, 2005-04-16 at 12:15 -0700, Linus Torvalds wrote:
>
> On Sat, 16 Apr 2005, Thomas Gleixner wrote:
> >
> > For the export stuff its terrible slow. :(
>
> What kind of _strange_ scripting architecture is so fast that there's a
> difference between "cat-file" and "ls-tree" and can handle 17,000 files in
> 60,000 revisions, yet so slow that you can't trivially convert 20 bytes of
> data?
Sorry I was neither talking about "cat-file ..." nor about the 20 byte
conversion. I was talking about the bk export script, which writes the
objects itself. Doing this with the git-tools would slow it down, as I
have the retrieved data already in memory. It does not slow me down to
create the binary ref, but its annoying.
I just figured, that some revtools might have the need to use direct
pointers into objects and face the same problem the other way round.
tglx
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Junio C Hamano @ 2005-04-16 19:27 UTC (permalink / raw)
To: Petr Baudis; +Cc: Thomas Gleixner, Linus Torvalds, Ingo Molnar, junkio, git
In-Reply-To: <20050416185751.GJ19099@pasky.ji.cz>
>>>>> "PB" == Petr Baudis <pasky@ucw.cz> writes:
PB> P.S.: It seems that Linus applied a patch to ls-tree which will make it
PB> read_sha1_file() on each item when ls-tree is recursive. Junio, why did
PB> you do it?
Sorry it was my misunderstanding, before I found out exactly how
S_ISDIR is used. Thank you for pointing it out.
I was confused by this comment around the area I changed:
/* XXX: We do some ugly mode heuristics here.
* It seems not worth it to read each file just to get this
* and the file size. -- pasky@ucw.cz
I mistakenly inferred from that comment that S_ISDIR(mode) is
not a guarantee. So I mistakenly optimized it for non-recursive
case by keeping that "heuristics". The logic was: If recursive
we will need to run read_sha1_file() to find out if it is really
a tree anyway.
I'll fix it up, now I know S_ISDIR(mode) is a guarantee that it
is a tree, I'll do the "heuristics" first, and do read_sha1_file
only when it is a tree and I am recursive.
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Junio C Hamano @ 2005-04-16 19:19 UTC (permalink / raw)
To: Mike Taht; +Cc: git
In-Reply-To: <42615B00.6090106@timesys.com>
>>>>> "MT" == Mike Taht <mike.taht@timesys.com> writes:
MT> alternatively, "git-archive-torrent" to create a list of files for a
MT> bittorrent feed....
That is certainly good for establishing the baseline, but you
still need to leverage the inherent delta-compressibility
between related blobs/trees by also doing something like what I
described as "diff package", don't you?
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Ingo Molnar @ 2005-04-16 19:18 UTC (permalink / raw)
To: David Mansfield; +Cc: git
In-Reply-To: <42612771.5000705@cobite.com>
* David Mansfield <david@cobite.com> wrote:
> Ingo Molnar wrote:
> >* Ingo Molnar <mingo@elte.hu> wrote:
> >
> >
> >>the patches contain all the existing metadata, dates, log messages and
> >>revision history. (What i think is missing is the BK tree merge
> >>information, but i'm not sure we want/need to convert them to GIT.)
> >
> >
> >author names are abbreviated, e.g. 'viro' instead of
> >viro@parcelfarce.linux.theplanet.co.uk, and no committer information is
> >included (albeit commiter ought to be Linus in most cases). These are
> >limitations of the BK->CVS gateway i think.
> >
>
> Glad to hear cvsps made it through! I'm curious what the manual
> fixups required were, except for the binary file issue (logo.gif).
--cvs-direct was needed to speed it up from 'several days to finish' to
'several hours to finish', but it crashed on a handful of patches [i
used the latest devel snapshot so this isnt a complaint]. (one of the
crashes was when generating 1860.patch.) Also, 'cvs rdiff' apparently
emits an empty patch for diffs that remove a file that end without
having a newline character - but this isnt cvsps's problem. (grep for
+++ in the patchset to find those cases.)
> As to the actual email addresses, for more recent patches, the
> Signed-off should help. For earlier ones, isn't their some script
> which 'knows' a bunch of canonical author->email mappings? (the
> shortlog script or something)?
yeah, that's not that much of a problem, most of the names are unique,
and the rest can be fixed up too.
> Is the full committer email address actually in the changeset in BK?
> If so, given that we have the unique id (immutable I believe) of the
> changset, could it be extracted directly from BK?
i think it's included in BK.
Ingo
^ permalink raw reply
* Re: SHA1 hash safety
From: Paul Jackson @ 2005-04-16 19:16 UTC (permalink / raw)
To: C. Scott Ananian; +Cc: omb, david.lang, mingo, git
In-Reply-To: <Pine.LNX.4.61.0504161040310.29343@cag.csail.mit.edu>
Scott wrote:
> Please, let's talk about hash collisions responsibly.
Agreed.
Chasing down links from the one Petr provided:
http://cryptography.hyperlink.cz/MD5_collisions.html
the best read I found was:
MD5 To Be Considered Harmful Someday
http://eprint.iacr.org/2004/357.pdf
As the author, Dan Kaminsky, states:
> it is far too easy to overestimate the risks described in this paper.
This paper does a good job of explaining the vulnerabilities
that MD5 has, currently (and yes, git uses SHA1 ...).
We have far greater vulnerabilities from intentional or accidental
coding errors, inadequately audited code, root exploits of user
(non-kernel) code, compilation and build tools, unreliable hardware
(how many of us use non-ECC memory - I do), poorly administered
systems, ...
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@engr.sgi.com> 1.650.933.1373, 1.925.600.0401
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Linus Torvalds @ 2005-04-16 19:15 UTC (permalink / raw)
To: Thomas Gleixner; +Cc: Ingo Molnar, git
In-Reply-To: <1113681021.28612.29.camel@tglx.tec.linutronix.de>
On Sat, 16 Apr 2005, Thomas Gleixner wrote:
>
> For the export stuff its terrible slow. :(
I don't really see your point.
If you already know what the tree is like you say, you don't care about
the tree object. And if you don't know what the tree is, what _are_ you
doing?
In other words, show us what you're complaining about. If you're looking
into the trees yourself, then the binary representation of the sha1 is
already what you want. That _is_ the hash. So why do you want it in ASCII?
And if you're not looking into the tree directly, but using "cat-file
tree" and you were hoping to see ASCII data, then that's certainly not
going to be any faster than just doing "ls-tree" instead.
In other words, I don't see your point. Either you want ascii output for
scripting, or you don't. First you claimed that you did, and that you
would want the tree object to change in order to do so. Now you claim that
you can't use "ls-tree" because it's too slow.
That just isn't making any sense. You're mixing two totally different
levels, and complaining about performance when scripting things. Yet
you're talking about a 20-byte data structure that is trivial to convert
to any format you want.
What kind of _strange_ scripting architecture is so fast that there's a
difference between "cat-file" and "ls-tree" and can handle 17,000 files in
60,000 revisions, yet so slow that you can't trivially convert 20 bytes of
data?
Linus
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Jan-Benedict Glaw @ 2005-04-16 19:13 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]
On Sat, 2005-04-16 10:04:31 -0700, Linus Torvalds <torvalds@osdl.org>
wrote in message <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>:
> What do people think? I'm not so much worried about the data itself: the
> git architecture is _so_ damn simple that now that the size estimate has
> been confirmed, that I don't think it would be a problem per se to put
> 3.2GB into the archive. But it will bog down "rsync" horribly, so it will
> actually hurt synchronization untill somebody writes the rev-tree-like
> stuff to communicate changes more efficiently..
>
> IOW, it smells to me like we don't have the infrastructure to really work
> with 3GB archives, and that if we start from scratch (2.6.12-rc2), we can
> build up the infrastructure in parallell with starting to really need it.
3GB is quite some data, but I'd accept and prefer to download it from
somewhere. I think that it's worth it.
I accept that there are people out there which would love to get a
smaller archive, but at least most developers that would actually use it
for day-to-day work *do* have the bandwidth to download it. Maybe we'd
also prepare (from time to time) bzip'ed tarballs, which I expect to be
a tad smaller.
MfG, JBG
--
Jan-Benedict Glaw jbglaw@lug-owl.de . +49-172-7608481 _ O _
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O
fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: Re: full kernel history, in patchset format
From: Petr Baudis @ 2005-04-16 18:57 UTC (permalink / raw)
To: Thomas Gleixner; +Cc: Linus Torvalds, Ingo Molnar, junkio, git
In-Reply-To: <1113681021.28612.29.camel@tglx.tec.linutronix.de>
Dear diary, on Sat, Apr 16, 2005 at 09:50:21PM CEST, I got a letter
where Thomas Gleixner <tglx@linutronix.de> told me that...
> On Sat, 2005-04-16 at 11:44 -0700, Linus Torvalds wrote:
>
> > That level of abstraction ("we never look directly at the objects") is
> > what allows us to change the object structure later. For example, we
> > already changed the "commit" date thing once, and the tree object has
> > obviously evolved a bit, and if we ever change the hash, the objects will
> > change too, but if you always just script them using nice helper tools,
> > you won't ever need to _care_. And that's how it should be.
>
> For the export stuff its terrible slow. :(
It seems to me that you must be doing something wrong then. I can't see
anything which would not make ls-tree blindingly fast (except for when
being recursive, see below).
BTW, what do you need ls-tree output for, when doing export _to_ git?
P.S.: It seems that Linus applied a patch to ls-tree which will make it
read_sha1_file() on each item when ls-tree is recursive. Junio, why did
you do it? Is there any possible case when the item would not be marked
as directory but it would be a tree object? I could imagine it bogging
down ls-tree on big tree a lot.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
^ permalink raw reply
* Re: Re: full kernel history, in patchset format
From: Christopher Li @ 2005-04-16 15:44 UTC (permalink / raw)
To: Petr Baudis; +Cc: Linus Torvalds, Ingo Molnar, git
In-Reply-To: <20050416174327.GG19099@pasky.ji.cz>
On Sat, Apr 16, 2005 at 07:43:27PM +0200, Petr Baudis wrote:
> Dear diary, on Sat, Apr 16, 2005 at 07:04:31PM CEST, I got a letter
> where Linus Torvalds <torvalds@osdl.org> told me that...
> > So I'd _almost_ suggest just starting from a clean slate after all.
> > Keeping the old history around, of course, but not necessarily putting it
> > into git now. It would just force everybody who is getting used to git in
> > the first place to work with a 3GB archive from day one, rather than
> > getting into it a bit more gradually.
> >
> > Comments?
>
> FWIW, it looks pretty reasonable to me. Perhaps we should have a
> separate GIT repository with the previous history though, and in the
> first new commit the parent could point to the last commit from the
> other repository.
>
> Just if it isn't too much work, though. :-)
I think we can make the git using stackable repository. When it fail
to find an object, it will try it's to read from parent repository.
It is useful to slice the history.
I can have local repository that all the new object create by me will
store in my tree instead of the official one. Clean up the object in the
my local tree will be much easier it only need to work on a much smaller
repository. If all my change is merge to official tree, I just simply
empty my local repository.
About the kernel git repository. I think it is much easier just put
them in one tree. So I don't need to worry about "if I need to see
pre 2.6.12, I need to do this". And the full repository need to
store in the server some where any way.
However I totally agree that people should not deal with unnecessary the history
when they start using the git tools. We should just make the tools
by default don't download all the histories. Only get it when user specific
ask for it.
Why 2.6.12-rc2? When kernel grows to 2.6.15, a new user might not even need
pre 2.6.13 most of the time. If we make it very easier for people to get
history if they need, it will make them less motivate to store unnecessary
history locally (just in case I need it).
I think we should not advise using rsync to sync the whole git tree as
way to get update. We need to get use to only have a slice of the history
and get more if we needed.
The server should should provide some small metadata file like the
the rev-tool cache, so the SCM tools can download it to figure out what file
is needed to download to get to certain revision. Instead of download the
whole repository to figure out what is new.
We can even slice that metadata information to smaller pieces base on major release point.
Chris
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Thomas Gleixner @ 2005-04-16 19:50 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504161135480.7211@ppc970.osdl.org>
On Sat, 2005-04-16 at 11:44 -0700, Linus Torvalds wrote:
> That level of abstraction ("we never look directly at the objects") is
> what allows us to change the object structure later. For example, we
> already changed the "commit" date thing once, and the tree object has
> obviously evolved a bit, and if we ever change the hash, the objects will
> change too, but if you always just script them using nice helper tools,
> you won't ever need to _care_. And that's how it should be.
For the export stuff its terrible slow. :(
I agree that using common tools is good. But we talk also about an open
format, so using a script to speed up certain tasks is not bad at all.
tglx
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Junio C Hamano @ 2005-04-16 18:46 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <7vmzry7ev5.fsf@assigned-by-dhcp.cox.net>
>>>>> "JCH" == Junio C Hamano <junkio@cox.net> writes:
JCH> I have been cooking this idea before I dove into the merge stuff
JCH> and did not have time to implement it myself (Hint Hint), but I
JCH> think something along the following lines would work nicely:
It should be fairly obvious from the context what I meant to
say, but in case somebody gets confused by my inaccurate
description of small details (or, before somebody nitpicks ;-),
I'd add some clarifications and corrections.
JCH> * Run diff-tree between neighboring commits [*1*] to find out
JCH> the set of blobs that are "related". Extract those related
JCH> blobs and run "diff" [*2*] between them to see if it produces
JCH> a patch smaller than the whole thing when compressed. If
JCH> diff+patch is a win, then we do not have to transmit the blob
JCH> that we could reproduce by sending the diff. Note that fact.
I talked only about blobs here, but I really mean all types:
commits, trees and blobs here. Nothing prevents us from
extracting the raw data for trees and commits and run diff
between them. We can use cat-file to do that today.
What we do not have is the reverse of "$ cat-file type >rawdata"
(i.e. "$ write-file type <rawdata"), but that is trivial to
write. The raw data for related tree objects should delta well.
I do not think it is worth the effort to attempt delta for
commit objects. Anything that git-archive-tar decides not to
send in diff+patch form, be it blob or tree or commit, should be
noted here, not just blob as my previous message incorrectly
implies.
JCH> Given the above, the operation of git-archive-patch is also
JCH> quite obvious. Extract the "diff package" tarball into the
JCH> objects/ directory that has (at least) the full Bn, uncompress
JCH> the patch file part, and run patch on it.
Of course after you ran patch to reproduce the raw data for the
blob or tree, we need the reverse of cat-file to register such
data under object/ hierarchy.
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Linus Torvalds @ 2005-04-16 18:44 UTC (permalink / raw)
To: Thomas Gleixner; +Cc: Ingo Molnar, git
In-Reply-To: <1113679421.28612.16.camel@tglx.tec.linutronix.de>
On Sat, 16 Apr 2005, Thomas Gleixner wrote:
>
> One remark on the tree blob storage format.
> The binary storage of the sha1sum of the refered object is a PITA for
> scripting.
> Converting the ASCII -> binary for the sha1sum comparision should not
> take much longer than the binary -> ASCII conversion for the file
> reference. Can this be changed ?
I'd really rather not. Why don't you just use "ls-tree" for scripting?
That's why it exists in the first place.
It might make sense to have some simple selection capabilities built into
ls-tree (ie "ls-tree --match drivers/char/ -z <treesha1>" to get just a
subtree out), but that depends entirely on how you end up using it.
The fact is, there should _never_ any reason to look at the objects
themselves directly. "cat-file" is a debugging aid, it shouldn't be
scripted (with the possible exception of "cat-file blob xxxx" to just
extract the blob contents, since that object doesn't have any internal
structure).
That level of abstraction ("we never look directly at the objects") is
what allows us to change the object structure later. For example, we
already changed the "commit" date thing once, and the tree object has
obviously evolved a bit, and if we ever change the hash, the objects will
change too, but if you always just script them using nice helper tools,
you won't ever need to _care_. And that's how it should be.
If there's a tool missing, holler. THAT is the part I've been trying to
write: all the plumbing so that you _can_ script the thing sanely, and not
worry about how objects are created and worked with.
For example, that "index" file format likely _will_ change. I ended up
doing the new "stage" flags in a way that kept the index file compatible
with old ones, but I did that mainly because it also happened to be the
easiest way to enforce the rule I wanted to enforce (ie the "stage" really
_is_ a part of the filename from a "compare filenames" standpoint, in
order to make sure that the stages are always ordered).
So if the index file change hadn't had that property, I'd have just said
"I'll change the format", and anybody who tried to parse the index file
would have been _broken_.
Linus
^ permalink raw reply
* Re: Re: full kernel history, in patchset format
From: Thomas Gleixner @ 2005-04-16 19:41 UTC (permalink / raw)
To: Petr Baudis; +Cc: Linus Torvalds, Ingo Molnar, git
In-Reply-To: <20050416183232.GH19099@pasky.ji.cz>
On Sat, 2005-04-16 at 20:32 +0200, Petr Baudis wrote:
> Dear diary, on Sat, Apr 16, 2005 at 09:23:40PM CEST, I got a letter
> where Thomas Gleixner <tglx@linutronix.de> told me that...
> > One remark on the tree blob storage format.
> > The binary storage of the sha1sum of the refered object is a PITA for
> > scripting.
> > Converting the ASCII -> binary for the sha1sum comparision should not
> > take much longer than the binary -> ASCII conversion for the file
> > reference. Can this be changed ?
>
> Huh, you aren't supposed to peek into trees directly. What's wrong with
> ls-tree?
Why I'm not supposed ? Is this evil ?
My export script has all the data available, so I write the tree refs
directly. The full export runs ~1 hour. Thats long enough :) I tried the
git way and it slows me down by factor "BIG" (I dont remember the
number)
Also for reference tracking all the information might be available e.g.
by a database. Why should the revtool then use some tool to retrieve
information which is already there ?
tglx
^ permalink raw reply
* Re: Re: Re: full kernel history, in patchset format
From: Petr Baudis @ 2005-04-16 18:36 UTC (permalink / raw)
To: Thomas Gleixner; +Cc: Linus Torvalds, Ingo Molnar, git
In-Reply-To: <20050416183232.GH19099@pasky.ji.cz>
Dear diary, on Sat, Apr 16, 2005 at 08:32:32PM CEST, I got a letter
where Petr Baudis <pasky@ucw.cz> told me that...
> Dear diary, on Sat, Apr 16, 2005 at 09:23:40PM CEST, I got a letter
> where Thomas Gleixner <tglx@linutronix.de> told me that...
> > One remark on the tree blob storage format.
> > The binary storage of the sha1sum of the refered object is a PITA for
> > scripting.
> > Converting the ASCII -> binary for the sha1sum comparision should not
> > take much longer than the binary -> ASCII conversion for the file
> > reference. Can this be changed ?
>
> Huh, you aren't supposed to peek into trees directly. What's wrong with
> ls-tree?
(I meant, you aren't supposed to peek into trees from scripts. Or well,
not "not supposed", but it does not make much sense when you have
ls-tree.)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Mike Taht @ 2005-04-16 18:35 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vmzry7ev5.fsf@assigned-by-dhcp.cox.net>
> * A script git-archive-tar is used to create a "base tarball"
> that roughly corresponds to "linux-*.tar.gz". This works as
> follows:
>
> $ git-archive-tar C [B1 B2...]
>
> This reads the named commit C, grabs the associated tree
> (i.e. its sub-tree objects and the blob they refer to), and
> makes a tarball of ??/??????????????????????????????????????
> files. The tarball does not have to contain any extra
> information to reproduce any ancestor of the named commit.
alternatively, "git-archive-torrent" to create a list of files for a
bittorrent feed....
--
Mike Taht
^ permalink raw reply
* Re: Re: full kernel history, in patchset format
From: Petr Baudis @ 2005-04-16 18:32 UTC (permalink / raw)
To: Thomas Gleixner; +Cc: Linus Torvalds, Ingo Molnar, git
In-Reply-To: <1113679421.28612.16.camel@tglx.tec.linutronix.de>
Dear diary, on Sat, Apr 16, 2005 at 09:23:40PM CEST, I got a letter
where Thomas Gleixner <tglx@linutronix.de> told me that...
> One remark on the tree blob storage format.
> The binary storage of the sha1sum of the refered object is a PITA for
> scripting.
> Converting the ASCII -> binary for the sha1sum comparision should not
> take much longer than the binary -> ASCII conversion for the file
> reference. Can this be changed ?
Huh, you aren't supposed to peek into trees directly. What's wrong with
ls-tree?
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Junio C Hamano @ 2005-04-16 18:31 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>
>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
LT> What do people think? I'm not so much worried about the data itself: the
LT> git architecture is _so_ damn simple that now that the size estimate has
LT> been confirmed, that I don't think it would be a problem per se to put
LT> 3.2GB into the archive. But it will bog down "rsync" horribly, so it will
LT> actually hurt synchronization untill somebody writes the rev-tree-like
LT> stuff to communicate changes more efficiently..
LT> IOW, it smells to me like we don't have the infrastructure to really work
LT> with 3GB archives, and that if we start from scratch (2.6.12-rc2), we can
LT> build up the infrastructure in parallell with starting to really need it.
LT> But it's _great_ to have the history in this format, especially since
LT> looking at CVS just reminded me how much I hated it.
LT> Comments?
I have been cooking this idea before I dove into the merge stuff
and did not have time to implement it myself (Hint Hint), but I
think something along the following lines would work nicely:
* A script git-archive-tar is used to create a "base tarball"
that roughly corresponds to "linux-*.tar.gz". This works as
follows:
$ git-archive-tar C [B1 B2...]
This reads the named commit C, grabs the associated tree
(i.e. its sub-tree objects and the blob they refer to), and
makes a tarball of ??/??????????????????????????????????????
files. The tarball does not have to contain any extra
information to reproduce any ancestor of the named commit.
When extra parameters, B1 B2..., are given, it also creates
"diff package" that roughly corresponds to "patch-*.gz" for
each Bn given. They must be ancestors of commit. The
intention is to store enough information to ensure that the
recipient can recreate all the SHA1 files "base tarball" for
commits between (Bn, C] would contain, provided if the
recipient already has all the SHA1 files "base tarball" for
Bn.
* A script git-archive-patch is used to read such a "diff
package".
So a user needs to:
* First pick some baseline B and download the base tarball for
commit B. It is up to him to make trade-offs between how far
back he wants to see the history and how much bandwidth he
wants to waste. Untar it to get the baseline.
* Then periodically pick up "diff package" for (C, B] where C
is the latest available. Run git-archive-patch to populate
the rest.
* In addition the user can run rsync with timestamp option to
pick up SHA1 files created upstream since C after this
happens.
What git-archive-tar needs to do to produce "diff package" for
(Bn, C] is fairly obvious.
* From rev-tree output, find all the commits that are on path
from Bn to C.
* Find all the SHA1 objects that appear on this commit chain;
subtract what is in Bn since we assume the recipient has them
already.
* Run diff-tree between neighboring commits [*1*] to find out
the set of blobs that are "related". Extract those related
blobs and run "diff" [*2*] between them to see if it produces
a patch smaller than the whole thing when compressed. If
diff+patch is a win, then we do not have to transmit the blob
that we could reproduce by sending the diff. Note that fact.
* When you are all done, you have a single patch file that
contains small edits on numerous blobs, and set of SHA1 files
that are cheaper to transmit than in the patch form.
Compress the patch file and package them together to make a
tar archive.
Given the above, the operation of git-archive-patch is also
quite obvious. Extract the "diff package" tarball into the
objects/ directory that has (at least) the full Bn, uncompress
the patch file part, and run patch on it.
[Footnotes]
*1* Alternatively, this diff-tree can be run between Bn and each
commit between (Bn, C]. It is like incremental dump strategy.
We should experiment and find a good balance.
*2* This does not have to be "diff -u" --- we are assuming the
exact patch so diff -e or xdelta would do. We should experiment
and find a good diff+patch pair.
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Thomas Gleixner @ 2005-04-16 19:23 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>
On Sat, 2005-04-16 at 10:04 -0700, Linus Torvalds wrote:
> So I'd _almost_ suggest just starting from a clean slate after all.
> Keeping the old history around, of course, but not necessarily putting it
> into git now. It would just force everybody who is getting used to git in
> the first place to work with a 3GB archive from day one, rather than
> getting into it a bit more gradually.
Sure. We can export the 2.6.12-rc2 version of the git'ed history tree
and start from there. Then the first changeset has a parent, which just
lives in a different place.
Thats the only difference to your repository, but it would change the
sha1 sums of all your changesets.
> What do people think? I'm not so much worried about the data itself: the
> git architecture is _so_ damn simple that now that the size estimate has
> been confirmed, that I don't think it would be a problem per se to put
> 3.2GB into the archive. But it will bog down "rsync" horribly, so it will
> actually hurt synchronization untill somebody writes the rev-tree-like
> stuff to communicate changes more efficiently..
We have all the tracking information in SQL and we will post the data
base dump soon, so people interested in revision tracking can use this
as an information base.
> But it's _great_ to have the history in this format, especially since
> looking at CVS just reminded me how much I hated it.
:)
One remark on the tree blob storage format.
The binary storage of the sha1sum of the refered object is a PITA for
scripting.
Converting the ASCII -> binary for the sha1sum comparision should not
take much longer than the binary -> ASCII conversion for the file
reference. Can this be changed ?
tglx
^ permalink raw reply
* Re: Re: full kernel history, in patchset format
From: Petr Baudis @ 2005-04-16 17:43 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>
Dear diary, on Sat, Apr 16, 2005 at 07:04:31PM CEST, I got a letter
where Linus Torvalds <torvalds@osdl.org> told me that...
> So I'd _almost_ suggest just starting from a clean slate after all.
> Keeping the old history around, of course, but not necessarily putting it
> into git now. It would just force everybody who is getting used to git in
> the first place to work with a 3GB archive from day one, rather than
> getting into it a bit more gradually.
>
> Comments?
FWIW, it looks pretty reasonable to me. Perhaps we should have a
separate GIT repository with the previous history though, and in the
first new commit the parent could point to the last commit from the
other repository.
Just if it isn't too much work, though. :-)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
^ permalink raw reply
* Re: space compression (again)
From: Martin Uecker @ 2005-04-16 17:37 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.61.0504161101470.29343@cag.csail.mit.edu>
[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]
On Sat, Apr 16, 2005 at 11:11:00AM -0400, C. Scott Ananian wrote:
> On Sat, 16 Apr 2005, Martin Uecker wrote:
>
> >The right thing (TM) is to switch from SHA1 of compressed
> >content for the complete monolithic file to a merkle hash tree
> >of the uncompressed content. This would make the hash
> >independent of the actual storage method (chunked or not).
>
> It would certainly be nice to change to a hash of the uncompressed
> content, rather than a hash of the compressed content, but it's not
> strictly necessary, since files are fetched all at once: there's not 'read
> subrange' operation on blobs.
>
> I assume 'merkle hash tree' is talking about:
> http://www.open-content.net/specs/draft-jchapweske-thex-02.html
> ..which is very interesting, but not quite what I was thinking.
> The merkle hash approach seems to require fixed chunk boundaries.
I don't know what is written there, but I don't
consider fixed chunk boundaries part of the definition.
> The rsync approach does not use fixed chunk boundaries; this is necessary
> to ensure good storage reuse for the expected case (ie; inserting a single
> line at the start or in the middle of the file, which changes all the
> chunk boundaries).
Yes. The chunk boundaries should be determined deterministically
from local properties of the data. Use a rolling checksum over
some small window and split the file it it hits a special value (0).
This is what the rsyncable patch to zlib does.
> Further, in the absence of subrange reads on blobs, it's not entirely
> clear what using a merkle hash would buy you.
The whole design of git is a hash tree. If you extend
this tree structure into files you end up with merkle
hash trees. Everything else is just more complicated.
Martin
--
One night, when little Giana from Milano was fast asleep,
she had a strange dream.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [PATCH 3/2] merge-trees script for Linus git
From: Junio C Hamano @ 2005-04-16 17:14 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0504160928250.7211@ppc970.osdl.org>
>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
LT> Anyway, with the modified read-tree, as far as I can tell it will now
LT> merge all the cases where one side has done something to a file, and the
LT> other side has left it alone (or where both sides have done the exact same
LT> modification). That should _really_ cut down the cases to just a few files
LT> for most of the kernel merges I can think of.
LT> Does it do the right thing for your tests?
Yes.
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Linus Torvalds @ 2005-04-16 17:04 UTC (permalink / raw)
To: Ingo Molnar; +Cc: git
In-Reply-To: <20050416131528.GB19908@elte.hu>
On Sat, 16 Apr 2005, Ingo Molnar wrote:
>
> i've converted the Linux kernel CVS tree into 'flat patchset' format,
> which gave a series of 28237 separate patches. (Each patch represents a
> changeset, in the order they were applied. I've used the cvsps utility.)
>
> the history data starts at 2.4.0 and ends at 2.6.12-rc2. I've included a
> script that will apply all the patches in order and will create a
> pristine 2.6.12-rc2 tree.
Hey, that's great. I got the CVS repo too, and I was looking at it, but
the more I looked at it, the more I felt that the main reason I want to
import it into git ends up being to validate that my size estimates are at
all realistic.
I see that Thomas Gleixner seems to have done that already, and come to a
figure of 3.2GB for the last three years, which I'm very happy with,
mainly because it seems to match my estimates to a tee. Which means that I
just feel that much more confident about git actually being able to handle
the kernel long-term, and not just as a stop-gap measure.
But I wonder if we actually want to actually populate the whole history..
Now that my size estimates have been verified, I have little actual real
reason to put the history into git. There are no visualization tools done
for git yet, and no helpers to actually find problems, and by the time
there will be, we'll have new history.
So I'd _almost_ suggest just starting from a clean slate after all.
Keeping the old history around, of course, but not necessarily putting it
into git now. It would just force everybody who is getting used to git in
the first place to work with a 3GB archive from day one, rather than
getting into it a bit more gradually.
What do people think? I'm not so much worried about the data itself: the
git architecture is _so_ damn simple that now that the size estimate has
been confirmed, that I don't think it would be a problem per se to put
3.2GB into the archive. But it will bog down "rsync" horribly, so it will
actually hurt synchronization untill somebody writes the rev-tree-like
stuff to communicate changes more efficiently..
IOW, it smells to me like we don't have the infrastructure to really work
with 3GB archives, and that if we start from scratch (2.6.12-rc2), we can
build up the infrastructure in parallell with starting to really need it.
But it's _great_ to have the history in this format, especially since
looking at CVS just reminded me how much I hated it.
Comments?
Linus
^ permalink raw reply
* git-pasky: gitXnormid.sh overhaul
From: Rene Scharfe @ 2005-04-16 16:51 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
Hello,
I just couldn't stand all the calls to grep and other external tools in
gitXnormid.sh and started rewriting it in a knee-jerk reaction.
You said in a private conversation that you don't like to include things
like ${var#stuff} to stay "sh compatible", while OTOH you favour $(cmd)
over `cmd`. Both are POSIX extensions of the classical Bourne Shell
syntax (see e.g. http://docs.hp.com/en/B2355-90046/ch15s03.html for a
feature comparision between POSIX shell, Bourne Shell and Korn Shells on
HP-UX). For reference, The Open Group publishes its IEEE Std 1003.1
standard (vulgo: POSIX) on this website:
http://www.opengroup.org/onlinepubs/009695399/toc.htm. So which shell
do you want to target with your git scripts?
This time I tested the script. :] It copes with invalid IDs,
non-existing valid IDs, abbreviated IDs, an omitted ID, valid IDs, with
tags and branch names. I also made sure the script runs with bash, ash,
pdksh, zsh and bash --posix (all on SuSE 9.2).
I changed the way an ID is verified. The script now tries to find tags
and branches first by looking for .git/tags/<id> and .git/HEAD.<id> and
after that looking inside .git/objects for a match. That's faster and
now I can safely give a branch a name consisting of 40 hex digits. :-)
The script follows in plain text format, not as a patch. Your and my
version share only very few lines, so this way it's easier to review.
I'll send you a patch if and when you're ready to apply it, ok?
Thanks,
Rene
--- 8< ---
#!/bin/sh
#
# Internal: Normalize the given ID to a tree ID.
# Copyright (c) Petr Baudis, 2005
#
# Takes an arbitrary ID as a parameter. -c tells it to give
# a commit id rather than tree id.
usage() {
echo "Usage: $0 [-c] [tree-id | commit-id | tag | branch]"
exit 2
}
get_first_word() {
if read one two; then
echo "$one"
return 0
fi
return 1
}
expand_hash() {
hashdir=${SHA1_FILE_DIRECTORY:-.git/objects}
filename=${1#??}
dirname=${1%${filename}}
first=true
for file in "${hashdir}/${dirname}/${filename}"*; do
[ -f "$file" ] || return 1
if $first; then
hash=${dirname}${file##*/}
first=false
else
return 1
fi
done
$first && return 1
echo "$hash"
}
get_tree_id() {
cat-file commit "$1" | while read tag hash; do
if [ "$tag" = "tree" ]; then
echo "$hash"
return
fi
done
}
type=tree
case "$1" in
-c) type=commit; shift;;
-*) usage;;
esac
if [ ! "$1" ]; then
if [ ! -f ".git/HEAD" ]; then
echo "$0: file .git/HEAD not found"
usage
fi
id=$(get_first_word <".git/HEAD")
elif [ -f ".git/tags/$1" ]; then
id=$(get_first_word <".git/tags/$1")
elif [ -f ".git/HEAD.$1" ]; then
id=$(get_first_word <".git/HEAD.$1")
else
id=$(expand_hash "$1")
fi
if [ $? != 0 ]; then
echo "$0: invalid ID: $1" >&2
exit 1
fi
if [ "$type" = "tree" ]; then
tree_id=$(get_tree_id "$id" 2>/dev/null)
[ "$tree_id" ] && id=$tree_id
fi
if [ $(cat-file -t "$id") != "$type" ]; then
echo "$0: invalid ID: $id" >&2
exit 1
fi
echo "$id"
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox