* Re: [RFC] Submodules in GIT
From: Linus Torvalds @ 2006-12-02 19:52 UTC (permalink / raw)
To: Josef Weidendorfer; +Cc: sf, Git Mailing List, Martin Waitz, Andy Parkins
In-Reply-To: <200612021232.08699.Josef.Weidendorfer@gmx.de>
On Sat, 2 Dec 2006, Josef Weidendorfer wrote:
> >
> > The only _true_ namespace would be the SHA1 of the commit (and maybe allow
> > a pointer to a tag too, but the namespace ends up being the same).
>
> I am not so sure about this.
> Perhaps we want the namespace to be more than the space of commit ids.
I don't think it would be wrong at all to have a "link object" type, and
have the "link" tree entry actually point to that "link object" instead of
pointing directly to the commit in the submodule.
And yes, that extra indirection would allow for more flexibility (the
"link object" can contain comments about the particular version used,
pointers to where you can get it - whether human-readable or strictly
meant for automation - etc etc).
So I agree with Andy Parkins' comment about the link object allowing not
only extended namespaces, but also allowing a certain amount of
flexibility (ie there's some built-in extensibility and ability to perhaps
add future fields if there's a new object type).
I just want the naming of the links themselves to use all the same SHA1
hashes etc, so that you always have a very explicit, and very trustworthy
version - and never end up in the situation that you know which repository
you want at that position, but you don't know exactly which commit in that
repo was supposed to be checked out with that particular version of the
super-module.
^ permalink raw reply
* Re: On removing files and "git-rm is pointless"
From: Sam Vilain @ 2006-12-02 20:00 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Carl Worth, git
In-Reply-To: <Pine.LNX.4.64.0612020919400.3476@woody.osdl.org>
Linus Torvalds wrote:
> I'd like it more if it defaulted to actually removing the file, preferably
> refusing to with an error message if the file didn't match the index.
index, or HEAD version? Otherwise you can "update-index"; "rm" without
seeing something wrong is happening.
> Final note: arguably, the current "git rm" is a better mirror image of
> "git add" than what I suggest above. "git add" doesn't actually create the
> working file (you had to do that yourself), so you _could_ argue that "git
> rm" as it stands now is closer to the "reverse" of git add. The same is
> true of the recursive behaviour.
For this reason I think that the current behaviour is not so broken.
Everywhere else, it is up to the user to make the changes to the working
copy that they want to commit. I like git-rm because I can go:
rm -rf whatever
git-rm whatever
I can see why you'd want
git-rm -u whatever
or
rm -rf whatever
git-commit -a
An extra flag to actually unlink the files is less likely to cause bugs
with porcelain expecting git-rm to behave as it does currently. If it
is to be changed in backwards incompatible ways, there should probably
be a deprecation time.
"rm -u" could alter the default semantics, ie, require the extra -r
option to recurse and require -f unless things are safe.
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:12 UTC (permalink / raw)
To: Linus Torvalds; +Cc: sf, sf, git
In-Reply-To: <Pine.LNX.4.64.0612011505190.3695@woody.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 979 bytes --]
hoi :)
On Fri, Dec 01, 2006 at 03:09:40PM -0800, Linus Torvalds wrote:
> On Fri, 1 Dec 2006, sf wrote:
> > If you do not want to fetch all of the supermodule then do not fetch the
> > supermodule.
>
> So why do you want to limit it? There's absolutely no cost to saying "I
> want to see all the common shared infrastructure, but I'm actually only
> interested in this one submodule that I work with".
An interesting way to support this "only fetch some modules" use-case is
to use several supermodules.
So you could have one supermodule which is geared towards developers and
only contains the modules they use. Another supermodule contails all
the toolchain sources. And then there is the supermodule used for
releases which is just a merge of all the other supermodules.
The concept is so flexible that you don't have to introduce lots of
other things as module namespaces. Just use the tools you have in a
creative way ;-)
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Test t4118 fails on Mac OS X due to dependency on GNU sed behaviour
From: Marco Roeland @ 2006-12-02 20:14 UTC (permalink / raw)
To: git
Hello list!
Recently test t4118 was added to prepare for upcoming changes in
behaviour of GNU diff.
This test fails on Mac OS X. It turns out that in the end the failure
comes from a difference in the behaviour between GNU sed and the version
that comes with OS X.
The feature that is tested works fine, but the implementation of the
examination of the result fails.
The difference is that the OS X version of sed always ends the last line
with a newline character, even if it wasn't there in the input. Strictly
speaking this is POSIX specified behaviour that was changed in the GNU
version of sed in 1997, citing from the changelog for the historically
curious minded people:
Mon Jul 28 10:50:41 PDT 1997 Ken Pizzini <ken@gnu.org>
* sed.c, sed.h, execute.c: POSIX.2, section 4.55.7, says that
a newline must end *every* output line. But I think
that it is useful (when seding a binary file) to omit
a trailing newline if the input lacks one. Thus the addition of
POSIXLY_CORRECT behavior.
* execute.c: however, when seding multiple files my feeling
is that it makes sense to have each file but the last
behave as-if it ended in a newline. Modified read_pattern_space()
accordingly.
A workaround on Mac OS X is easy by for example just installing the GNU
version of sed from the 'fink' archive with "sudo apt-get install sed".
Funny enough the system 'diff' is already the GNU version, so the test
t4118 itself might even be necessary for upcoming feline versions of
OS X!
Unfortunately I myself don't see an elegant and portable fix in the test
itself... [hint]
--
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Jakub Narebski @ 2006-12-02 20:16 UTC (permalink / raw)
To: git
In-Reply-To: <20061120215116.GA20736@admingilde.org>
Few thoughts on this topic. Some of those are repeating what
was said eaelier
1. Submodule (subproject) as commit-in-a-tree
Let's try to paint a little diagram (attribution missing):
belonging to:
/--------- supermodule -------\ /---- submodule -------\
commit -> tree +-> blob
| +-> tree -> ...
| +-----------------> commit -> tree -> ...
v |
commit -> tree +-> ... v
| +-----------------> commit -> ...
v / |
commit -> tree +-> ... / |
| +---------/ v
| commit -> ...
v |
commit -> tree +-> ... v
+-----------------> commit
Both have their independent history, but they are linked as some
submodule versions are part of the supermodule tree.
2. Working area for project with submodules
Submodule as separate repository model
supermodule
+ .git/ <------------------------.
+ HEAD |
+ index |
+ objects/ |
+ objects/info/alternates ---. |
+ subdir1/ | |
+ sub1file | ^
+ submodule/ | |
+ .git v |
+ HEAD | |
+ index | |
+ objects/ <--------------' |
+[objects/info/borrowers] ----'
+ subsubdir/
+ submfile
+ file
Embedded submodule model
supermodule
+ .git/
+ HEAD
+ index
+ objects/
+[refs/submodules/submodule/HEAD]
+[refs/submodules/submodule/index]
+ subdir1/
+ sub1file
+ submodule/
+ subsubdir/
+ submfile
+ file
The [fictional] borrowers file is for git-prune and friends (also
git-repack with -d option) to not remove objects needed by supermodule
(when for example submodule history got rewritten). But you can do
without it, as long as you don't rewind or don't prune in
supermodule.
The problem with submodule as separate git repository is that if you
move submodule (subproject) somewhere else in the repository (or just
rename it), you have to update alternates file... and this happens not
only on move itself, but also on checkout and reset. But that can be
managed by having in alternates all possible places the submodule ends
into. I don't know if it is truly a problem.
Alternate solution would be to have submodule objects [also] in the
main (superproject) object database (for example fetched from
submodule object repository on supermodule commit with changing
submodule).
Perhaps instead of objects/info/alternates we should use
objects/info/modules, or even modules file (as top .git dir).
The problem with embedded submodule model is ensuring that changes in
submodule go to submodule (using submodule refs; at least HEAD and
submodule index). And there are troubles with treating submodule
separately, for example cloning submodule only, or fetching from
submodule only.
3. Output of git-ls-tree and git-ls-files (git-ls-index ;-) for
project with submodules.
$ git ls-tree HEAD
040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 subdir1
140000 subm ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7 submodule
100644 blob a57a33b81ac6c9cb5ec0c833edc21bd66428d976 file
$ git ls-tree -r -t HEAD
040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 subdir1
100644 blob 70d8b9838a7333bc5a1edb93cf0e9abdbcf146cc subdir1/sub1file
140000 subm ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7 submodule
040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 submodule/subsubdir
100755 blob 6579f06b05c91f00f4f45015894f2bfab1076bf6 submodule/subsubdir/submfile
100644 blob a57a33b81ac6c9cb5ec0c833edc21bd66428d976 file
$ git ls-files --stages
100644 70d8b9838a7333bc5a1edb93cf0e9abdbcf146cc 0 subdir1/sub1file
140000 ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7 0 submodule
100644 a57a33b81ac6c9cb5ec0c833edc21bd66428d976 0 file
4. Workflow(s) for project with submodules
$ cd submodule
submodule$ edit subsubdir/submfile
submodule$ git update-index subsubdir/submfile # this updates submodule index
submodule$ git commit -m "Submodule change" # this changes submodule HEAD
submodule$ cd ..
$ git update-index submodule # this updates index
to submodule HEAD version
$ git commit -m "Change in submodule" # this updates HEAD
Of course as usual you should be able to do "git commit -a" to skip
"git update-index". One has to remember that "git update-index
submodule" and "git commit submodule" uses HEAD version of submodule,
not the working area version.
There was an idea to update superproject index not to HEAD version
but to some specified branch version.
5. Extended sha1 syntax for submodules
For [almost] all commands the commit-in-tree should
be viewed as tree-ish, for example in HEAD:submodule/subsubdir (is a
tree), or HEAD:submodule/subsubdir/submfile (is a blob).
Currently a suffix ':' followed by a path names the blob or tree (or
commit) at the given path in the tree-ish object named by the part
before the colon. You cannot currently use it recirsively, i.e. use
<tree-ish>:<path> to refer to tree (or commit), and use ':' after
that, e.g. <tree-ish>:<path>:<subpath>... well, currently this has not
much sense, as you can (and have to) use '/' as a separator.
There was proposal to use '//' as a way to force commit object in the
tree to be treated as commit-ish, not as a tree, so you can apply all
the extended sha1 machinery suitable for commits like ^, ^n, ~n and
also probably ^@, but perhaps not @{n}. Then making ':' resursive
would be useful, for example:
HEAD^:submodule//~2:subsubdir/submfile
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:18 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Josef Weidendorfer, sf, git
In-Reply-To: <Pine.LNX.4.64.0612011540010.3695@woody.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]
hoi :)
On Fri, Dec 01, 2006 at 04:12:10PM -0800, Linus Torvalds wrote:
> It only gets interesting for commands that fetch new objects, ie do a
> "pull/fetch" op, and you'd need to know where/how to fetch new objects for
> the xyzzy subproject, so that's a "naming" issue. You have a few choices:
>
> - get all the objects directly from the subproject as if it was one big
> project.
>
> I actually think this sucks. Why? Because it puts an insane load on the
> server side, which basically needs to traverse the object list of the
> _sum_ of all projects. An initial clone (or a really big pull, which
> comes to the same thing) would be absolutely horrendous
I don't buy your scalability argument.
By dividing the object traversal in separate steps you do not win
anything. The complexity of the operation still stays the same, as you
still have to traverse the exact same amount of objects.
By separating the repositories you just make reachability analyis be
totally awkward, without winning anything.
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:21 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Josef Weidendorfer, sf, Git Mailing List, Andy Parkins
In-Reply-To: <Pine.LNX.4.64.0612021144520.3476@woody.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 861 bytes --]
hoi :)
On Sat, Dec 02, 2006 at 11:52:13AM -0800, Linus Torvalds wrote:
> I don't think it would be wrong at all to have a "link object" type, and
> have the "link" tree entry actually point to that "link object" instead of
> pointing directly to the commit in the submodule.
>
> And yes, that extra indirection would allow for more flexibility (the
> "link object" can contain comments about the particular version used,
> pointers to where you can get it - whether human-readable or strictly
> meant for automation - etc etc).
What makes a submodule so special that now we suddenly have to store
those stuff in the object database?
Storing a fetch location would grossly contradict the distributed nature
of git. I really do not see _any_ reason to store more information than
the commit sha1 of the submodule.
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:24 UTC (permalink / raw)
To: Josef Weidendorfer; +Cc: Linus Torvalds, sf, git
In-Reply-To: <200612020017.44275.Josef.Weidendorfer@gmx.de>
[-- Attachment #1: Type: text/plain, Size: 695 bytes --]
hoi :)
On Sat, Dec 02, 2006 at 12:17:44AM +0100, Josef Weidendorfer wrote:
> After some thinking, a submodule namespace even is important for checking
> out only parts of a supermodule, exactly because the root of a submodule
> potentially can change at every commit.
have you ever thought about the idea that the location may be an
important thing to consider for your decision.
Perhaps the submodule is now used for something else (this is why it was
moved) and that now you'd like to keep it?
Anyway, you can just create several supermodules or implement generic
partial tree support for git. I do not see any reason to special case
submodules here.
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:40 UTC (permalink / raw)
To: Andy Parkins; +Cc: git
In-Reply-To: <200612021004.22236.andyparkins@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1781 bytes --]
hoi :)
On Sat, Dec 02, 2006 at 10:04:20AM +0000, Andy Parkins wrote:
> On Friday 2006, December 01 22:08, Martin Waitz wrote:
>
> > > echo $SUBMODULE_HASH >
> > > submodule/.git/refs/supermodules/commit$SUPERMODULE_HASH
> >
> > I guess you are aware that you have to scan _all_ trees inside _all_
> > supermodule commits for possible references.
>
> No you don't; you do it as part of the appropriate normal operations.
>
> * supermodule commit - scan the current tree for "link" objects in the
> tree. If you find one write the reference in the submodule.
> * adding a new submodule - if this is a new submodule there can't be any
> references in the supermodule already.
> * cloning a supermodule, every new commit that gets written in the
> supermodule gets checked from "link" objects.
* removing a branch from the supermodule.
OK, this is an infrequent operation and it can be handled by redoing
everything.
I just don't like to duplicate information which is already available
easily. We'd need much to many special cases, just to correctly support
reachablility analysis.
KISS.
> > So what do you do with deleted submodules?
> > You wouldn't want them to still sit around in your working directory,
> > but you still have to preserve them.
>
> Now that is a tricky one. Mind you, I think that problem exists for any
> implementation. I haven't got a good answer for that.
If you just keep it in a shared object repository you don't have any
problems.
Please note that it is not required to keep it in one physical location.
You can still use alternates/whatever to store some objects in another
repository, but you need to be able to access all objects from the
supermodule.
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:43 UTC (permalink / raw)
To: Josef Weidendorfer; +Cc: Andy Parkins, git
In-Reply-To: <200612021450.46005.Josef.Weidendorfer@gmx.de>
[-- Attachment #1: Type: text/plain, Size: 922 bytes --]
hoi :)
On Sat, Dec 02, 2006 at 02:50:45PM +0100, Josef Weidendorfer wrote:
> On Saturday 02 December 2006 11:04, Andy Parkins wrote:
> > > So what do you do with deleted submodules?
> > > You wouldn't want them to still sit around in your working directory,
> > > but you still have to preserve them.
> >
> > Now that is a tricky one. Mind you, I think that problem exists for any
> > implementation. I haven't got a good answer for that.
>
> That suggests that it is probably better to separate submodule repositories
> from their checked out working trees. Why not put the GITDIRs of the submodules
> in subdirectories of the supermodules GITDIR instead?
Why not simply use a shared object database instead?
You can still have an alternative to some standalone bare repository of
the submodule if you do not like to store submodule objects in the
supermodule repository.
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Linus Torvalds @ 2006-12-02 20:44 UTC (permalink / raw)
To: Martin Waitz; +Cc: Josef Weidendorfer, sf, git
In-Reply-To: <20061202201826.GR18810@admingilde.org>
On Sat, 2 Dec 2006, Martin Waitz wrote:
>
> I don't buy your scalability argument.
Try it.
Really. Get the mozilla import (450MB project), and clone it on a machine
with half a gig of RAM or less.
Then, clone a couple of smaller archives that end up being 450MB
_combined_, but clone them separately.
And watch the memory usage.
> By separating the repositories you just make reachability analyis be
> totally awkward, without winning anything.
Trust me. Try it out.
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Linus Torvalds @ 2006-12-02 20:46 UTC (permalink / raw)
To: Martin Waitz; +Cc: Josef Weidendorfer, sf, Git Mailing List, Andy Parkins
In-Reply-To: <20061202202103.GS18810@admingilde.org>
On Sat, 2 Dec 2006, Martin Waitz wrote:
>
> What makes a submodule so special that now we suddenly have to store
> those stuff in the object database?
I'm not sure it is. I suspect a pure commit link with just a CVS-style
"modules" file is sufficient. I'm just saying that I don't think it is
_wrong_ to possibly want to expand it.
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 20:58 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Josef Weidendorfer, sf, Git Mailing List, Andy Parkins
In-Reply-To: <Pine.LNX.4.64.0612021245081.3476@woody.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 911 bytes --]
hoi :)
On Sat, Dec 02, 2006 at 12:46:01PM -0800, Linus Torvalds wrote:
> On Sat, 2 Dec 2006, Martin Waitz wrote:
> >
> > What makes a submodule so special that now we suddenly have to store
> > those stuff in the object database?
>
> I'm not sure it is. I suspect a pure commit link with just a CVS-style
> "modules" file is sufficient. I'm just saying that I don't think it is
> _wrong_ to possibly want to expand it.
If we later see that we really want to have it we can always introduce
it later. I don't think we should do it now if we don't see clear
benefits _now_.
So I was not against the link object itself (initially I wanted to do it
this way, too), only agains the information which was proposed to be
stored there. Up to now I haven't found anything which makes sense to
store next to the submodule commit to define the identity of the
submodule.
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: On removing files and "git-rm is pointless"
From: Horst H. von Brand @ 2006-12-02 20:59 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Carl Worth, git
In-Reply-To: <Pine.LNX.4.64.0612020919400.3476@woody.osdl.org>
Linus Torvalds <torvalds@osdl.org> wrote:
[...]
> One thing to look out for: "git rm" actually defaults to the recursive
> behaviour, something that might take people by surprise. If you give it a
> directory name, it will happily delete all tracked files from within that
> directory, even without "-r". That is probably a design mistake. So it
> would probably make sense to:
>
> - without "-r", don't do the partial matches at the beginning (but still
> do globbing matches, of course, so "git rm dir/*" wouldn't need an
> "-r", but "git rm -r dir/", which does the same thing, _would_ need an
> "-r" to be effective)
The command should never even see the '*' here. Globbing is handled
(uniformly) by the shell. Don't loose that.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 2654431
Universidad Tecnica Federico Santa Maria +56 32 2654239
^ permalink raw reply
* Re: [RFC] Introduce "git stage" (along with some heresy)
From: Sam Vilain @ 2006-12-02 21:03 UTC (permalink / raw)
To: Carl Worth; +Cc: git
In-Reply-To: <87slfzfri7.wl%cworth@cworth.org>
Carl Worth wrote:
> Change #2: Make a staged commit an explicit act
> ===============================================
> The "-a" stands out to me here as the only command-line option needed
> in the first list, and the only command in the second list that
> performs a staged operation by default. So change number to is to
> redefine "commit" to mean what "commit -a" meant before and to require
> a new command-line option for staged committing, (the best naming I
> have so far is "commit --staged" with a shortcut of "commit -i"---the
> mismatch of "'i' as short for --staged" is a bit unlovely I admit).
I wonder about backwards compatibility, but then another part of me says
that porcelain are probably using "git-commit-tree" anyway.
How about considering alternative words? Like "git save" for this
higher level and more user friendly interface.
As another idea (brainstorming here), what about an "autocommit" approach?
git rm # removes files and asks for commit message
git add # ditto
git commit # updates and commits everything
git stage # starts a staged commit
git add # modifies staging area
git rm # ditto
git stage filename # adds contents to staging area
git commit # saves staging area as commit
Then you could have "core.autocommit" as a repo-config option,
defaulting to off for "backwards compatibility".
> Change #3: Change "add" to not stage any content
> ================================================
> To finish off, I'd like to propose descriptions of the commands to
> allow the user to use the "without staging" commands as a complete set
> while being able to easily ignore any of the staging capabilities.
> This does trigger a need for a semantic change in the "add"
> command. Here are the proposed descriptions:
The "autocommit" concept may make this less of an issue.
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Martin Waitz @ 2006-12-02 21:06 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Josef Weidendorfer, sf, git
In-Reply-To: <Pine.LNX.4.64.0612021242080.3476@woody.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 658 bytes --]
hoi :)
On Sat, Dec 02, 2006 at 12:44:20PM -0800, Linus Torvalds wrote:
> On Sat, 2 Dec 2006, Martin Waitz wrote:
> >
> > I don't buy your scalability argument.
>
> Try it.
>
> Really. Get the mozilla import (450MB project), and clone it on a machine
> with half a gig of RAM or less.
>
> Then, clone a couple of smaller archives that end up being 450MB
> _combined_, but clone them separately.
>
> And watch the memory usage.
Do I understand you correctly that the problem is not the algorithmic
complexity but that you have to map the objects at once instead of map
them in small parts one after the other?
--
Martin Waitz
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: selective git-update-index per diff(1) chunks
From: Sam Vilain @ 2006-12-02 21:08 UTC (permalink / raw)
To: Peter Baumann; +Cc: git, Alexey Dobriyan
In-Reply-To: <slrnen04os.a5.Peter.B.Baumann@xp.machine.xx>
Peter Baumann wrote:
>> Now I want to commit them as two separate commits. So far, it's
>> * getting full diff
>> * cp(1)
>> * hand-edit both diffs
>> * commit first
>> * commit second
I made a patch to 'cg-commit -p' to do this, check the archives.
>> Relying on diff(1) definition of chunks is sorta hacky, though... I admit it.
>
> I don't think it belongs in the plumbing, the git-update-index but I
> think something like this would be very usefull.
>
> AFAIR darcs has this functionality. It selectively ask for each hunk if
> it should be commited. This would be awfull to have in git.
darcs and SVK both have this "interactive commit".
darcs also does something very useful with this; what amounts to
automatically making topic branches. Currently there is no easy way to
manage that, either.
Sam.
^ permalink raw reply
* Re: On removing files and "git-rm is pointless"
From: Sam Vilain @ 2006-12-02 21:10 UTC (permalink / raw)
To: Horst H. von Brand; +Cc: Linus Torvalds, Carl Worth, git
In-Reply-To: <200612022059.kB2Kx565009646@laptop13.inf.utfsm.cl>
Horst H. von Brand wrote:
>> - without "-r", don't do the partial matches at the beginning (but still
>> do globbing matches, of course, so "git rm dir/*" wouldn't need an
>> "-r", but "git rm -r dir/", which does the same thing, _would_ need an
>> "-r" to be effective)
>
> The command should never even see the '*' here. Globbing is handled
> (uniformly) by the shell. Don't loose that.
But;
git-rm \*>/dev/null; git-add \*
is just so darned handy!
^ permalink raw reply
* Re: [RFC] Submodules in GIT
From: Linus Torvalds @ 2006-12-02 21:22 UTC (permalink / raw)
To: Martin Waitz; +Cc: Josef Weidendorfer, sf, git
In-Reply-To: <Pine.LNX.4.64.0612021242080.3476@woody.osdl.org>
On Sat, 2 Dec 2006, Linus Torvalds wrote:
>
> And watch the memory usage.
Btw, just in case you don't understand _why_ this is true, the fact is, in
a git repository, quite fundamentally, because we don't have "backlinks"
at any stage at all, we don't know - and fundamentally _cannot_ know -
whether we're goign to see the same object in the future.
So operations like "git-rev-list --objects" (or, these days, more commonly
anything that just does the equivalent of that internally using the
library interfaces - ie "git pack-objects" and friends) VERY FUNDAMENTALLY
have to hold on to the object flags for the whole lifetime of the whole
operation.
And you should realize that this is really really fundamental. You can't
fix it with "smarter memory management". You can't fix it with "garbage
collection". This is _not_ a result of the fact that we use C and malloc,
and we don't free those objects, like some people sometimes seem to
believe.
So garbage collection will never help this kind of situation. It flows
_directly_ from the fact that our objects are immutable: because they are
immutable, they don't have any backpointers, because we cannot (and must
not) add backpointers to an old existing object when a new object is
created that points to it.
So this really isn't a memory management issue. You could somewhat work
around it by adding a "caching layer" on top of git, and allow that
caching layer to modify their cache of old objects (so that they can
contain back-pointers), but for 99% of all users that would actually make
performance MUCH WORSE, and it would also be a serious problem for
coherency issues (one of the things that immutable objects cause is that
there are basically never any race conditions, while a "caching layer"
like this would have some serious issues about serialization).
So: the very fundamental nature and choices that were made in git also
means that when you have something like git-pack-objects that wants to
walk the whole repo, you will end up with something that remembers EVERY
SINGLE OBJECT it walked.
And while I've worked very hard to make the memory footprint of individual
objects as small as possible, and this means that this all works fine even
for fairly large databases (especially since very few operations actually
do this "traverse the whole friggin tree" thing), it does mean that
there's a very fundamental limit to scalability. You can't just make a
whole repository a hundred times bigger - because the operations that
traverse the whole thing will require a hundred times more memory!
Now, in "real" projects, this is not a problem. I can pretty much
_guarantee_ that memory sizes and hardware will grow faster than projects
grow. I'm not AT ALL worried about the fact that in ten years, the linux
kernel repository will likely be two or three times the size it is now.
Because I'm absolutely convinced that in ten years, the machines we have
now will be obsolete.
So on any "individual project" basis, the fact that memory requirements
scale roughly as O(n) in the total repository size is simply not a
problem. In fact, O(n) is pretty damn good, especially since the constant
is pretty small (basically 28 bytes per object - and 20 of those bytes
are the SHA1 that you simply cannot avoid).
But it does mean that supermodules really should NOT be so seamless that
doing a "git clone" on a supermodule does one _large_ clone. Because it's
simply going to be better to:
- when you clone the supermodule, track the commits you need on all
submodules (this _may_ be a reason in itself for the "link" object,
just so that you can traverse the supermodule object dependencies and
know what subobject you are looking at even _without_ having to look at
the path you got there from)
- clone submodules one-by-one, using the list of objects you gathered.
Maybe there are other solutions, but quite frankly, I doubt it. Yes,
you'll end up "traversing" exactly as many objects either way, but the
"globe subobjects one by one" is going to be a _hell_ of a lot more
memory-efficient, and quite frankly, "memory usage" == "performance"
under many loads (notably, any load that uses too much memory will _suck_
performance-wise, either because of swapping or simply because it will
throw out caches that "many small invocations" would not have thrown out).
So I guarantee that it's going to be better to do five clones of five
small repositories over one clone of one big one. If only because you need
less memory to do the five smaller clones.
^ permalink raw reply
* Patch: git-add --exclude=pattern
From: Adam J. Richter @ 2006-12-02 21:29 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]
git-ls-files has an argument "--exclude=<pattern>". I have
copied the relevant code snippets to do the same for git-add-files.
I want this facility so that I can disable processing of
.gitignore files for some shell scripts that I've cobbled together
to update git repositories from tar files of new versions of software
packages. Adding "!*" to .git/info/exclude does not achieve this,
because the .gitignore files have priority, and I can see arguments
why this should be so, so that .git/info/exclude can just encode
defaults and individual directories can have more specific rules.
In comparison, passing "--exclude='!*'" apparently does have priority
over the .gitignore files.
By the way, the particular situation where I had this problem
was in linux-2.6.19, which shoots itself in the foot a bit by including
.gitignore files that exclude source files like arch/*/kernel/vmlinux.ld.S,
include/asm-*/{alternative-asm,frame}.i and the .cvsignore files themselves.
However, fixing this linux kernel problem is not the point of this email.
I want a facility that can reliably add files from source trees even if
they have minor bugs like this in their .gitignore rules.
Many thanks to ShadeHawk on the git IRC channel for pointing
out to me that the git-find-ls manual page actually describes the problem
and a fix for the case of vmlinux.ld.S in the linux kernel tree, and for
suggesting --exclude=pattern instead of --exclude-per-directory=filename.
Adam Richter
[-- Attachment #2: add-exclude.diff --]
[-- Type: text/plain, Size: 1187 bytes --]
diff --git a/src/builtin-add.c b/src/builtin-add.c
index febb75e..a501b50 100644
--- a/src/builtin-add.c
+++ b/src/builtin-add.c
@@ -11,7 +11,7 @@
#include "cache-tree.h"
static const char builtin_add_usage[] =
-"git-add [-n] [-v] <filepattern>...";
+"git-add [--exclude=<pattern>] [-n] [-v] <filepattern>...";
static void prune_directory(struct dir_struct *dir, const char **pathspec, int prefix)
{
@@ -55,7 +55,6 @@ static void fill_directory(struct dir_st
int baselen;
/* Set up the default git porcelain excludes */
- memset(dir, 0, sizeof(*dir));
dir->exclude_per_dir = ".gitignore";
path = git_path("info/exclude");
if (!access(path, R_OK))
@@ -90,6 +89,7 @@ int cmd_add(int argc, const char **argv,
const char **pathspec;
struct dir_struct dir;
+ memset(&dir, 0, sizeof(dir));
git_config(git_default_config);
newfd = hold_lock_file_for_update(&lock_file, get_index_file(), 1);
@@ -106,6 +106,10 @@ int cmd_add(int argc, const char **argv,
i++;
break;
}
+ if (!strncmp(arg, "--exclude=", 10)) {
+ add_exclude(arg+10, "", 0, &dir.exclude_list[EXC_CMDL]);
+ continue;
+ }
if (!strcmp(arg, "-n")) {
show_only = 1;
continue;
^ permalink raw reply related
* Re: [RFC] Submodules in GIT
From: Linus Torvalds @ 2006-12-02 21:29 UTC (permalink / raw)
To: Martin Waitz; +Cc: Josef Weidendorfer, sf, git
In-Reply-To: <20061202210640.GX18810@admingilde.org>
On Sat, 2 Dec 2006, Martin Waitz wrote:
>
> Do I understand you correctly that the problem is not the algorithmic
> complexity but that you have to map the objects at once instead of map
> them in small parts one after the other?
Not map them, but track their "used" flag. Yes. You can unmap objects any
time at all (since you can just always re-create them at any time very
easily and cheaply), but the one thing you CANNOT recreate is the object
flags. See "struct object", and the "used" and FLAG_BITS in particular.
Almost all git programs need the FLAG_BITS. Something as simple as just
traversing the commit history needs at a minimum one _single_ bit for each
object: "Have I already seen this". In reality, you tend to need two or
three more (ie the UNINTERESTING bit ends up being as important as the
SEEN bit, because it's what determines whether it's reachable from some
commit we're _not_ interested in, and in the end that's what allows us to
not traverse the whole history).
So you need at a MINIMUM to track the bits
#define SEEN (1u<<0)
#define UNINTERESTING (1u<<1)
and in practice almost everything needs
#define SHOWN (1u<<3)
too (SEEN is for deciding whether to _traverse_ something, SHOWN is for
deciding whether you've already output the data for this, and the
difference is crucial for any depth-first DAG algorithm, since you need
to test-and-set the one bit when you first encounter the object, and
test-and-set the other bit when you "leave" the object).
So three bits are minimal to _any_ git traversal algorithm. Many specific
issues want more bits (eg the TREECHANGE bit may not be quite as
fundamnetal, but it sure ends up being critical for the "track subtree"
case).
^ permalink raw reply
* Request: Adding -q/--quiet option to git-cat-file
From: Jakub Narebski @ 2006-12-02 21:32 UTC (permalink / raw)
To: git
The only "committag" gitweb implements now is linking the full 40-char sha
of a commit to a "commit" view. This requires checking the type of object
given by sha1... but git-cat-file -t writes errors to stdout if the object
does not exist. I'd rather not use additional call, using git-cat-file -e,
or using 'string' version of magic "-|" open to redirect stdout to /dev/null
(as it is additional fork anyway).
What I need is an option -q/--quiet to git-cat-file to forbid printing error
messages to stderr... but it looks like it would be easier to add it
universally for all commands, or add it to git wrapper.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: On removing files and "git-rm is pointless"
From: Linus Torvalds @ 2006-12-02 21:33 UTC (permalink / raw)
To: Horst H. von Brand; +Cc: Carl Worth, git
In-Reply-To: <200612022059.kB2Kx565009646@laptop13.inf.utfsm.cl>
On Sat, 2 Dec 2006, Horst H. von Brand wrote:
>
> The command should never even see the '*' here. Globbing is handled
> (uniformly) by the shell. Don't loose that.
Git supprts an extra level of globbing.
In the kernel tree, try the difference between these two command lines:
git ls-files fs/*.c
git ls-files 'fs/*.c'
and enjoy.
It's _very_ useful for a number of programs (including "git add") but I
find it _especially_ useful for something like
git grep jiffies -- '*.h'
where the git-level globbign is important.
Not all git programs do that globbing, but many do.
^ permalink raw reply
* git-svn: File added multiple times?
From: Florian Weimer @ 2006-12-02 21:38 UTC (permalink / raw)
To: git
Is this expected?
$ sort /tmp/git-svn-output | uniq -c | sort -rn | head
4 A mlton/trunk/doc/web/papers/index.html
4 A mlton/trunk/doc/web/papers/01-icfp.ps.gz
4 A mlton/trunk/doc/web/papers/00-esop.ps.gz
4 A mlton/trunk/doc/examples/save-world/save-world.sml
4 A mlton/trunk/doc/examples/save-world/Makefile
4 A mlton/trunk/doc/examples/profiling/profiling.sml
4 A mlton/trunk/doc/examples/profiling/Makefile
4 A mlton/trunk/doc/examples/ffi/Makefile
4 A mlton/trunk/doc/examples/ffi/main.sml
4 A mlton/trunk/doc/examples/ffi/ffi.h
$
It's somewhat counter-intuitive, at least. This is with Debian's
^ permalink raw reply
* Re: [PATCH] git-svn: correctly access repos when only given partial read permissions
From: Sam Vilain @ 2006-12-02 21:57 UTC (permalink / raw)
To: Eric Wong; +Cc: Junio C Hamano, git
In-Reply-To: <11644366982320-git-send-email-normalperson@yhbt.net>
Eric Wong wrote:
> Sometimes users are given only read access to a subtree inside a
> repository, and git-svn could not read log information (and thus
> fetch commits) when connecting a session to the root of the
> repository. We now start an SVN::Ra session with the full URL
> of what we're tracking, and not the repository root as before.
>
> This change was made much easier with a cleanup of
> repo_path_split() usage as well as improving the accounting of
> authentication batons.
This broke mirroring file:/// URIs;
eg, if I have ~/.svk/local as a SVN repository, which has a complete
mirror of a URL under mirror/fai, and I want to copy the revisions into
git using git-svn, I use:
perl ~/src/git/git-svn multi-init -t tags -T trunk \
file:///home/samv/.svk/local/mirror/fai
I now get this error:
Filesystem has no item: File not found: revision 8514, path
'/mirror/fai/tags/mirror/fai/tags' at /home/samv/src/git/git-svn line 3236
(next, I'll make git-svn correctly look at the svm:* revprops to get the
upstream repo URL and revision number for the commit message)
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox