* Re: [PATCH] rebase -i: commit when continuing after "edit"
From: Johannes Sixt @ 2007-09-25 15:54 UTC (permalink / raw)
To: David Kastrup; +Cc: git, Johannes Schindelin
In-Reply-To: <86ve9y6bvm.fsf@lola.quinscape.zz>
David Kastrup schrieb:
> Johannes Sixt <j.sixt@viscovery.net> writes:
>> I poked around a bit, but one major obstacle is that the assignments
>> in $author_script are on separate lines, which you would have to
>> splice into a single line before you can insert them in the eval.
>
> Hm? Why? Newlines separate assignments just as reliable as spaces
> do. They are primarily special to the tty as line separators, not the
> shell as such.
The task here is to have the assignments on the same line as the command at
the end so that they are locally exported. Here we are inside an 'eval', and
the new-lines *do* what their name suggest: make new lines.
-- Hannes
^ permalink raw reply
* Re: [PATCH] user-manual: Explain what submodules are good for.
From: J. Bruce Fields @ 2007-09-25 16:09 UTC (permalink / raw)
To: Michael Smith; +Cc: Miklos Vajna, git
In-Reply-To: <1190724278-8586-1-git-send-email-msmith@cbnco.com>
On Tue, Sep 25, 2007 at 08:44:38AM -0400, Michael Smith wrote:
> Rework the introduction to the Submodules section to explain why
> someone would use them, and fix up submodule references from the
> tree-object and todo sections.
Looks good to me; thanks!
Acked-by: J. Bruce Fields <bfields@citi.umich.edu>
--b.
> Signed-off-by: Michael Smith <msmith@cbnco.com>
> ---
> Documentation/user-manual.txt | 54 +++++++++++++++++++++++++++++++---------
> 1 files changed, 42 insertions(+), 12 deletions(-)
>
> diff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt
> index a085ca1..c7fdf25 100644
> --- a/Documentation/user-manual.txt
> +++ b/Documentation/user-manual.txt
> @@ -2856,8 +2856,7 @@ between two related tree objects, since it can ignore any entries with
> identical object names.
>
> (Note: in the presence of submodules, trees may also have commits as
> -entries. See gitlink:git-submodule[1] and gitlink:gitmodules.txt[1]
> -for partial documentation.)
> +entries. See <<submodules>> for documentation.)
>
> Note that the files all have mode 644 or 755: git actually only pays
> attention to the executable bit.
> @@ -3163,12 +3162,45 @@ information as long as you have the name of the tree that it described.
> Submodules
> ==========
>
> -This tutorial explains how to create and publish a repository with submodules
> -using the gitlink:git-submodule[1] command.
> -
> -Submodules maintain their own identity; the submodule support just stores the
> -submodule repository location and commit ID, so other developers who clone the
> -superproject can easily clone all the submodules at the same revision.
> +Large projects are often composed of smaller, self-contained modules. For
> +example, an embedded Linux distribution's source tree would include every
> +piece of software in the distribution with some local modifications; a movie
> +player might need to build against a specific, known-working version of a
> +decompression library; several independent programs might all share the same
> +build scripts.
> +
> +With centralized revision control systems this is often accomplished by
> +including every module in one single repository. Developers can check out
> +all modules or only the modules they need to work with. They can even modify
> +files across several modules in a single commit while moving things around
> +or updating APIs and translations.
> +
> +Git does not allow partial checkouts, so duplicating this approach in Git
> +would force developers to keep a local copy of modules they are not
> +interested in touching. Commits in an enormous checkout would be slower
> +than you'd expect as Git would have to scan every directory for changes.
> +If modules have a lot of local history, clones would take forever.
> +
> +On the plus side, distributed revision control systems can much better
> +integrate with external sources. In a centralized model, a single arbitrary
> +snapshot of the external project is exported from its own revision control
> +and then imported into the local revision control on a vendor branch. All
> +the history is hidden. With distributed revision control you can clone the
> +entire external history and much more easily follow development and re-merge
> +local changes.
> +
> +Git's submodule support allows a repository to contain, as a subdirectory, a
> +checkout of an external project. Submodules maintain their own identity;
> +the submodule support just stores the submodule repository location and
> +commit ID, so other developers who clone the containing project
> +("superproject") can easily clone all the submodules at the same revision.
> +Partial checkouts of the superproject are possible: you can tell Git to
> +clone none, some or all of the submodules.
> +
> +The gitlink:git-submodule[1] command is available since Git 1.5.3. Users
> +with Git 1.5.2 can look up the submodule commits in the repository and
> +manually check them out; earlier versions won't recognize the submodules at
> +all.
>
> To see how submodule support works, create (for example) four example
> repositories that can be used later as a submodule:
> @@ -3213,8 +3245,8 @@ The `git submodule add` command does a couple of things:
>
> - It clones the submodule under the current directory and by default checks out
> the master branch.
> -- It adds the submodule's clone path to the `.gitmodules` file and adds this
> - file to the index, ready to be committed.
> +- It adds the submodule's clone path to the gitlink:gitmodules[5] file and
> + adds this file to the index, ready to be committed.
> - It adds the submodule's current commit ID to the index, ready to be
> committed.
>
> @@ -4277,5 +4309,3 @@ Write a chapter on using plumbing and writing scripts.
> Alternates, clone -reference, etc.
>
> git unpack-objects -r for recovery
> -
> -submodules
> --
> 1.5.3
>
^ permalink raw reply
* Re: [PATCH] rebase -i: commit when continuing after "edit"
From: David Kastrup @ 2007-09-25 16:04 UTC (permalink / raw)
To: git
In-Reply-To: <46F92F26.6050204@viscovery.net>
Johannes Sixt <j.sixt@viscovery.net> writes:
> David Kastrup schrieb:
>> Johannes Sixt <j.sixt@viscovery.net> writes:
>>> I poked around a bit, but one major obstacle is that the assignments
>>> in $author_script are on separate lines, which you would have to
>>> splice into a single line before you can insert them in the eval.
>>
>> Hm? Why? Newlines separate assignments just as reliable as spaces
>> do. They are primarily special to the tty as line separators, not the
>> shell as such.
>
> The task here is to have the assignments on the same line as the
> command at the end so that they are locally exported. Here we are
> inside an 'eval', and the new-lines *do* what their name suggest: make
> new lines.
The documentation to eval clearly states:
`eval'
eval [ARGUMENTS]
The arguments are concatenated together into a single command,
which is then read and executed, and its exit status returned as
the exit status of `eval'. If there are no arguments or only
empty arguments, the return status is zero.
So we are talking about a single command here. However, we indeed get
$ eval "echo x
> y
> z"
x
bash: y: command not found
bash: z: command not found
$
Um, so I have been talking nonsense. Does "the docs made me do it"
count as excuse?
--
David Kastrup
^ permalink raw reply
* Re: diffcore-rename performance mode
From: Jeff King @ 2007-09-25 16:38 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
In-Reply-To: <20070918085413.GA11751@coredump.intra.peff.net>
On Tue, Sep 18, 2007 at 04:54:13AM -0400, Jeff King wrote:
> > > However, keeping around _just_ the
> > > cnt_data caused only about 100M of extra memory consumption (and gave
> > > the same performance boost).
> >
> > That would be an interesting and relatively low-hanging optimization.
>
> I can produce memory usage numbers for the kernel, too.
And here are some kernel numbers. I measured performance of this script
in the linux-2.6 repository:
#!/bin/sh
last=
git-tag | grep -v -- - | while read tag; do
if test -n "$last"; then
echo Diffing $last..$tag
git-diff --raw -M -l0 $last $tag >/dev/null
fi
last=$tag
done
under the assumption that diffing between major revisions would give a
good medium of diffs that would be large enough to show the n^2 rename
behavior, but still small enough to be close to "everyday" usage.
I measured three different approaches:
1. stock 'next' (stock)
2. removing entirely the calls to diff_free_filespec_data (nofree)
3. changing those free calls to free everything except cnt_data (somefree)
And I measured two things:
1. user CPU time to complete
2. peak memory usage
All numbers are warm-cache, and typical cases after multiple runs.
| stock | nofree | somefree
-----------------|---------------------------
user time (s) | 76.78 | 16.96 | 46.26
peak memory (Kb) | 52300 | 66796 | 59156
The raw 'time' output is below:
stock:
76.78user 3.35system 1:20.72elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+166733minor)pagefaults 0swaps
nofree:
16.96user 1.46system 0:18.47elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+185353minor)pagefaults 0swaps
somefree:
46.26user 1.54system 0:47.94elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+178819minor)pagefaults 0swaps
So this is definitely worth pursuing, as it yields massive speedups even
for regular repositories. And even the 'nofree' case only costs us 14M
of extra memory (although it is a 27% increase, this just isn't that
memory-hungry an endeavour for the sizes of changes we're talking
about). And as Linus noted, now that we have a default rename limit,
you're not likely to hit an explosion of memory usage.
What is most confusing is why the 'somefree' case performs so badly,
since we should just be using the cnt_data. I'll see if gprof can shed
any light on that. It would be nice to use it instead, since it will
have much better memory usage in the face of large blobs (e.g., my
pathological case that started this whole thread).
-Peff
^ permalink raw reply
* Re: [PATCH] post-checkout hook, and related docs and tests
From: Josh England @ 2007-09-25 16:41 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vfy138vql.fsf@gitster.siamese.dyndns.org>
On Mon, 2007-09-24 at 16:54 -0700, Junio C Hamano wrote:
> "Josh England" <jjengla@sandia.gov> writes:
>
> > On Mon, 2007-09-24 at 14:07 -0700, Junio C Hamano wrote:
> > ...
> >> If you want to spacial case
> >>
> >> $ git checkout otherbranch path.c
> >>
> >> it raises another issue. Which commit should supply the
> >> "extended attribute description" for path.c? Should it be taken
> >> from the current commit (aka HEAD), otherbranch, or the index?
> >
> > This already is a special case and your question is valid but not one
> > that git should necessary care about. Since extended attributes are not
> > built into git the only way to handle them is through hooks. A such, it
> > is up to the hook to worry about these kinds of issues.
>
> The fear I have is that that kind of thinking would necessitate
> your hook to be called after the user edits paths.c in any other
> way not to confuse users.
>
> What I am questioning is where we should stop, in order to keep
> things simpler to explain, and I happen to think that it is far
> easier if we can teach that "git checkout other path.c" is
> equivalent to "git cat-file blob other:path.c >path.c" followed
> by "git add path.c", than saying "checkout is magical and if you
> have external hook it can do far more than editing the file
> yourself to arrive at the same contents".
>
> But I am obviously not the one who is interested in tracking
> extended attributes attached to git contents, and I do not feel
> too strongly about one way or the other. I am Ok with it if you
> think "checkout is magical" is easier to teach [*1*].
>
> I just wanted to make sure we know what semantics this is
> bringing in, and get it clearly documented. That's all.
OK. I'll try to come up with some good wording for the documentation.
So this leads to my next question: Should the post-merge patch be
brought in under this same umbrella to form a single post-checkout hook,
or should it stay a separate hook?
-JE
^ permalink raw reply
* Workflow question
From: Russ Brown @ 2007-09-25 16:43 UTC (permalink / raw)
To: git
Hi,
I've been trying to think of a git workflow that we could use to replace
our current svn/svk setup without simply using git in exactly the same
way that we use svn/svk.
Basically, we develop, maintain and enhance a website. On the central
repo is trunk which represents live, and any number of project branches.
Developers don't use local branches: they check out the project branches
they're working on and commit to those. Developers merge from trunk to
project branch from time to time to keep them current, and when a
project rolls out the branch is merged to trunk.
In addition to the obvious advantages that git would give us (such as
properly tracking that code author as opposed to the person who did the
merge), I'm wanting to gain the following benefits:
* The repository is very large (multiple gigabytes) and mirroring using
svk obviously takes a lot of time and space, so I'm keen to bring that
down, most likely by the developer not needing to mirror branches he
doesn't care about, or by being able to throw away branches he's done with.
* The repository is full of revisions that fail review (or break
things) and are fixed by subsequent revisions. We'd much rather be able
to have the developer fix his revisions before they get committed
'upstream' (whatever that ends up meaning).
I asked earlier about the email-based model that git itself uses, and
while it appears to work very well for a widely-dispersed open-source
project, I think it will be too cumbersome and slow for a fast-paced
internal development team who make a number of live releases every day.
So, I've been thinking and have come up with this, which I'd appreciate
comments about:
1. On a server we stick a git repository which contains the master
branch, which represents what trunk did (i.e. the live platform). This
branch contains the full history for the live platform.
2. On the same server we clone that repo to create a second repository
which is the developer area. In this we track master from the live repo,
and also create project branches.
3. Developers clone this developer repo, but I'd like them to be able
to decide which branches they actually want to clone from that
repository rather than simply cloning them all. Is this possible?
4. Developers create a local branch of the project they
are working on and commit to that.
5. Once they think they're done, they publish their branch to the
development repo and request for comments.
6. If all is not well, the developer creates a new local branch and
moves good revisions from his previous one to the new one, modifying
things as he goes, and republishes his new branch.
7. If all is well, their works gets merged or rebased onto the main
project branch, and once that's ready it gets pushed to the master and
rolled to live. The developer's individual branches get deleted from the
dev repo since they're no longer required.
8. From time to time the master branch gets merged to the project
branches. Developer's local branches can be rebased against the project
branch as they please.
Firstly, is all of this possible, and if so would it be considered a
good way of going about it?
Any comments appreciated.
--
Russ
^ permalink raw reply
* Re: [PATCH] resend: really plug memory leaks in git-svnimport
From: Junio C Hamano @ 2007-09-25 17:55 UTC (permalink / raw)
To: Stefan Sperling; +Cc: git
In-Reply-To: <20070924105740.GB8900@ted>
Stefan Sperling <stsp@elego.de> writes:
> Junio asked me to resend this patch to the mailing list.
>
> This version of the patch is adjusted to apply cleanly
> to current HEAD.
>
> @Junio: I'm not resending the multiple branch/tag dirs patch
> just yet, because I want to polish it first -- I've got another
> idea how to improve it.
Ok.
People on the list who still use git-svnimport, could you help
with testing this patch? Will queue for 'pu' in the meantime.
^ permalink raw reply
* Re: [PATCH] resend: really plug memory leaks in git-svnimport
From: Andreas Ericsson @ 2007-09-25 18:37 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Stefan Sperling, git
In-Reply-To: <7vr6km6354.fsf@gitster.siamese.dyndns.org>
Junio C Hamano wrote:
> Stefan Sperling <stsp@elego.de> writes:
>
>> Junio asked me to resend this patch to the mailing list.
>>
>> This version of the patch is adjusted to apply cleanly
>> to current HEAD.
>>
>> @Junio: I'm not resending the multiple branch/tag dirs patch
>> just yet, because I want to polish it first -- I've got another
>> idea how to improve it.
>
> Ok.
>
> People on the list who still use git-svnimport, could you help
> with testing this patch? Will queue for 'pu' in the meantime.
>
I used to use it, but I've given up on it in favour of git svn.
Partly because git-svn seems to get cases right that git-svnimport
didn't, but mostly because it remembers where I fetched from, which
is damn handy.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: diffcore-rename performance mode
From: Jeff King @ 2007-09-25 19:06 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
In-Reply-To: <20070925163843.GA22987@coredump.intra.peff.net>
On Tue, Sep 25, 2007 at 12:38:43PM -0400, Jeff King wrote:
>[...]
>
> What is most confusing is why the 'somefree' case performs so badly,
> since we should just be using the cnt_data. I'll see if gprof can shed
OK, I found the problem. estimate_similarity calls
diff_populate_filespec each time, even if we already have the cnt_data,
which leads to recomputing the blob contents from deltas. Oops.
Fixing this, the correct numbers are:
| stock | nofree | old somefree | fixed somefree
-----------------|-----------------------------------------------
user time (s) | 76.78 | 16.96 | 46.26 | 16.99
peak memory (Kb) | 52300 | 66796 | 59156 | 57328
So now we're at a 4.5x speedup for about 10% extra memory usage. Patch
will follow.
-Peff
^ permalink raw reply
* Re: Workflow question
From: Andreas Ericsson @ 2007-09-25 19:09 UTC (permalink / raw)
To: Russ Brown; +Cc: git
In-Reply-To: <46F93A99.5080707@gmail.com>
Russ Brown wrote:
> Hi,
>
> I've been trying to think of a git workflow that we could use to replace
> our current svn/svk setup without simply using git in exactly the same
> way that we use svn/svk.
>
> Basically, we develop, maintain and enhance a website. On the central
> repo is trunk which represents live, and any number of project branches.
> Developers don't use local branches: they check out the project branches
> they're working on and commit to those. Developers merge from trunk to
> project branch from time to time to keep them current, and when a
> project rolls out the branch is merged to trunk.
>
> In addition to the obvious advantages that git would give us (such as
> properly tracking that code author as opposed to the person who did the
> merge), I'm wanting to gain the following benefits:
>
> * The repository is very large (multiple gigabytes) and mirroring using
> svk obviously takes a lot of time and space, so I'm keen to bring that
> down, most likely by the developer not needing to mirror branches he
> doesn't care about, or by being able to throw away branches he's done with.
> * The repository is full of revisions that fail review (or break
> things) and are fixed by subsequent revisions. We'd much rather be able
> to have the developer fix his revisions before they get committed
> 'upstream' (whatever that ends up meaning).
>
> I asked earlier about the email-based model that git itself uses, and
> while it appears to work very well for a widely-dispersed open-source
> project, I think it will be too cumbersome and slow for a fast-paced
> internal development team who make a number of live releases every day.
>
We came to the same conclusion at our workplace. Email works great, but
it's faster and better to just walk over to your colleague and ask what
he/she thinks about something.
> So, I've been thinking and have come up with this, which I'd appreciate
> comments about:
>
> 1. On a server we stick a git repository which contains the master
> branch, which represents what trunk did (i.e. the live platform). This
> branch contains the full history for the live platform.
A must-have for any more-than-two-developers setup, so so far so good ;-)
> 2. On the same server we clone that repo to create a second repository
> which is the developer area. In this we track master from the live repo,
> and also create project branches.
This isn't necessary. Branches in git are very nearly zero-cost, so having
them in the same repo as the master branch won't hurt a bit. You can add
an update-hook on the mothership repo to restrict access to the master
branch if you like, but creating two separate repos will likely give
more headache than it's worth.
> 3. Developers clone this developer repo, but I'd like them to be able
> to decide which branches they actually want to clone from that
> repository rather than simply cloning them all. Is this possible?
Yes, although I'd actually recommend you to clone the full repo anyway.
Since the various branches are likely to share quite a lot of history
the added overhead of a few extra branches will most likely be negligible.
git makes even very large codebases appear small and unobtrusive. The
linux kernel history since 2.6.12 contains 554853 objects and compresses
down to 178MiB.
I think KDE is the largest repo imported to git so far. I've forgotten
the exact numbers, but everyone was very impressed, and quite surprised,
at the vast difference between SVN and git storage requirements.
> 4. Developers create a local branch of the project they
> are working on and commit to that.
> 5. Once they think they're done, they publish their branch to the
> development repo and request for comments.
Using topic-branches is a much better strategy, usually, since that
allows each feature to be evaluated and improved on on its own, rather
than having to merge *all* of a particular developers changes just to
get desirable feature X. Note that cherry-pick provides ways of doing
that anyways, but in a much less elegant way, and your integrator/
release engineer will likely tear his hair out on a daily basis without
topic branches.
> 6. If all is not well, the developer creates a new local branch and
> moves good revisions from his previous one to the new one, modifying
> things as he goes, and republishes his new branch.
> 7. If all is well, their works gets merged or rebased onto the main
> project branch, and once that's ready it gets pushed to the master and
> rolled to live. The developer's individual branches get deleted from the
> dev repo since they're no longer required.
Topic branches would work the same, basically, except they can be pushed up
for review a lot faster.
If all the pushing gets cumbersome, it also makes it easy to send the patches
out as emails for discussion. It's usually easier to let git handle the
actual code transmissions, but discussing patches in emails works quite
well if it's intended for a wider audience.
> 8. From time to time the master branch gets merged to the project
> branches. Developer's local branches can be rebased against the project
> branch as they please.
>
criss-cross merging can turn kinda nasty though, as you may have a hard time
finding *the* common point when you run into that rogue merge with conflict
markers everywhere (it happens for everyone sooner or later).
I'd suggest you rebase the developer/topic branches onto master with regular
intervals instead.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: diffcore-rename performance mode
From: Andreas Ericsson @ 2007-09-25 19:10 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20070925190640.GA4613@coredump.intra.peff.net>
Jeff King wrote:
> On Tue, Sep 25, 2007 at 12:38:43PM -0400, Jeff King wrote:
>
>> [...]
>>
>> What is most confusing is why the 'somefree' case performs so badly,
>> since we should just be using the cnt_data. I'll see if gprof can shed
>
> OK, I found the problem. estimate_similarity calls
> diff_populate_filespec each time, even if we already have the cnt_data,
> which leads to recomputing the blob contents from deltas. Oops.
>
> Fixing this, the correct numbers are:
>
> | stock | nofree | old somefree | fixed somefree
> -----------------|-----------------------------------------------
> user time (s) | 76.78 | 16.96 | 46.26 | 16.99
> peak memory (Kb) | 52300 | 66796 | 59156 | 57328
>
> So now we're at a 4.5x speedup for about 10% extra memory usage. Patch
> will follow.
>
Nice work :)
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* [PATCH] Remove 'submodules' from the TODO section of the User Manual.
From: Miklos Vajna @ 2007-09-25 19:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
A chapter about this topic has been added recently.
Signed-off-by: Miklos Vajna <vmiklos@frugalware.org>
---
i forgot to include this change in my previous patch, so here is the
correction.
Documentation/user-manual.txt | 2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt
index a085ca1..e4b04d0 100644
--- a/Documentation/user-manual.txt
+++ b/Documentation/user-manual.txt
@@ -4277,5 +4277,3 @@ Write a chapter on using plumbing and writing scripts.
Alternates, clone -reference, etc.
git unpack-objects -r for recovery
-
-submodules
--
1.5.3.2.80.g077d6f-dirty
^ permalink raw reply related
* Re: [RFC] Convert builin-mailinfo.c to use The Better String Library.
From: Steven Burns @ 2007-09-25 19:19 UTC (permalink / raw)
To: git
In-Reply-To: <86odfstbc6.fsf@lola.quinscape.zz>
The C++ community in general suffers a lot from the NIH Syndrome.
Matrixes, Strings, Vectors, everybody creates their own which are always, or
course, superior to what's already available.
Again, is not the language's fault, a language is just a language.
It's the way it has been driven.
My two cents.
"David Kastrup" <dak@gnu.org> wrote in message
news:86odfstbc6.fsf@lola.quinscape.zz...
> figo <rcc_dark@hotmail.com> writes:
>
>> http://www.research.att.com/~bs/applications.html
>>
>> just as Bjarne once wrote in his TC++PL, its hard to teach an old dog new
>> tricks. Its even harder to give quality education about how to use
>> something
>> to someone who doesnt want to learn.
>>
>> you hate high level, then continue programming operative systems,
>> please NEVER DO something else. C++ was designed to give programmers
>> high level tools and still being able to take care about
>> performance.
>>
>> portability wont be possible after a standard is published and some
>>couple of years given to the compiler developers. C++ had its
>>standard in 1998, and add two or three years for compiler development
>>= 2002. "Quite recently", way more recently that your last use of C++
>>I can bet.
>
> Care to explain why there are still not two numerical C++ libraries
> with compatible matrix classes?
>
> What use is talking about portability and high level when a basic
> interoperability feature that has been available since the sixties
> (more than 4 decades ago) in Fortran has not yet managed to make it
> into C++? C++ by now more or less offers a (somewhat deficient)
> standardized way to work with complex numbers, but matrices are still
> not standardized in any manner, and libraries won't interoperate.
>
> So C++ should get its head wrapped around the _low_ level problems
> first. It is a bloody shame that it still has not caught up with
> Fortran IV (or even Fortran II) with regard to usefulness for
> numerical libraries.
>
> It is not a matter of "hating high level" to see that C++ is mostly
> focused about addressing the wrong kinds of problems in the wrong
> ways. The pain/gain ratio is just bad.
>
> --
> David Kastrup
>
^ permalink raw reply
* Re: [PATCH] Remove 'submodules' from the TODO section of the User Manual.
From: J. Bruce Fields @ 2007-09-25 19:23 UTC (permalink / raw)
To: Miklos Vajna; +Cc: Junio C Hamano, git
In-Reply-To: <20070925191159.GG18370@genesis.frugalware.org>
On Tue, Sep 25, 2007 at 09:11:59PM +0200, Miklos Vajna wrote:
> A chapter about this topic has been added recently.
Thanks, but it looks like Michael Smith already included this in his
latest patch:
http://marc.info/?l=git&m=119072428615724&w=2
--b.
^ permalink raw reply
* [PATCH] diffcore-rename: cache file deltas
From: Jeff King @ 2007-09-25 19:29 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
We find rename candidates by computing a fingerprint hash of
each file, and then comparing those fingerprints. There are
inherently O(n^2) comparisons, so it pays in CPU time to
hoist the (rather expensive) computation of the fingerprint
out of that loop (or to cache it once we have computed it once).
Previously, we didn't keep the filespec information around
because then we had the potential to consume a great deal of
memory. However, instead of keeping all of the filespec
data, we can instead just keep the fingerprint.
This patch implements and uses diff_free_filespec_data_large
to accomplish that goal. We also have to change
estimate_similarity not to needlessly repopulate the
filespec data when we already have the hash.
Practical tests showed 4.5x speedup for a 10% memory usage
increase.
Signed-off-by: Jeff King <peff@peff.net>
---
The implementation is a little less nice than I would like, but I was
trying to be non-invasive. Specifically:
- the name diff_free_filespec_data_large is horrible, but this is based
on the fact that diff_free_filespec_data actually does too much (it
frees the data _and_ some other auxiliary data). And renaming that
would entail changing many callsites.
- It seems that a better place to call diffcore_populate_filespec
(rather than in estimate_similarity) would actually be in
diffcore_count_changes (when we _know_ that we need to populate the
contents data).
- The hash_chars() should arguably be tied into
diffcore_populate_filespec, which should have more of a "what
information do you want" interface. I.e., the "size_only" parameter
could be extended to a bitfield to say "populate this, and I need the
delta fingerprint, size, actual contents, etc". Then callers could
just use "populate" before looking at the filespec and it would
lazily load whatever they needed.
This patch cuts my pathological case from 20 minutes to 2 minutes, which
is a great improvement, but still unusable. However, now I should be
able to get more useful numbers on what else can be sped up.
diff.c | 7 ++++++-
diffcore-rename.c | 7 ++++---
diffcore.h | 1 +
3 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/diff.c b/diff.c
index 2216d75..647bcf0 100644
--- a/diff.c
+++ b/diff.c
@@ -1654,7 +1654,7 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
return 0;
}
-void diff_free_filespec_data(struct diff_filespec *s)
+void diff_free_filespec_data_large(struct diff_filespec *s)
{
if (s->should_free)
free(s->data);
@@ -1665,6 +1665,11 @@ void diff_free_filespec_data(struct diff_filespec *s)
s->should_free = s->should_munmap = 0;
s->data = NULL;
}
+}
+
+void diff_free_filespec_data(struct diff_filespec *s)
+{
+ diff_free_filespec_data_large(s);
free(s->cnt_data);
s->cnt_data = NULL;
}
diff --git a/diffcore-rename.c b/diffcore-rename.c
index 41b35c3..4fc2000 100644
--- a/diffcore-rename.c
+++ b/diffcore-rename.c
@@ -184,7 +184,8 @@ static int estimate_similarity(struct diff_filespec *src,
if (base_size * (MAX_SCORE-minimum_score) < delta_size * MAX_SCORE)
return 0;
- if (diff_populate_filespec(src, 0) || diff_populate_filespec(dst, 0))
+ if ((!src->cnt_data && diff_populate_filespec(src, 0))
+ || (!dst->cnt_data && diff_populate_filespec(dst, 0)))
return 0; /* error but caught downstream */
@@ -377,10 +378,10 @@ void diffcore_rename(struct diff_options *options)
m->score = estimate_similarity(one, two,
minimum_score);
m->name_score = basename_same(one, two);
- diff_free_filespec_data(one);
+ diff_free_filespec_data_large(one);
}
/* We do not need the text anymore */
- diff_free_filespec_data(two);
+ diff_free_filespec_data_large(two);
dst_cnt++;
}
/* cost matrix sorted by most to least similar pair */
diff --git a/diffcore.h b/diffcore.h
index eef17c4..4bf175b 100644
--- a/diffcore.h
+++ b/diffcore.h
@@ -48,6 +48,7 @@ extern void fill_filespec(struct diff_filespec *, const unsigned char *,
extern int diff_populate_filespec(struct diff_filespec *, int);
extern void diff_free_filespec_data(struct diff_filespec *);
+extern void diff_free_filespec_data_large(struct diff_filespec *);
extern int diff_filespec_is_binary(struct diff_filespec *);
struct diff_filepair {
--
1.5.3.2.1061.gc056e-dirty
^ permalink raw reply related
* Re: Workflow question
From: Jeff King @ 2007-09-25 19:34 UTC (permalink / raw)
To: Andreas Ericsson; +Cc: Russ Brown, git
In-Reply-To: <46F95CCC.4080209@op5.se>
On Tue, Sep 25, 2007 at 09:09:00PM +0200, Andreas Ericsson wrote:
> We came to the same conclusion at our workplace. Email works great, but
> it's faster and better to just walk over to your colleague and ask what
> he/she thinks about something.
One of the projects I am working on does things this way, but I have to
admit that I miss the email code-review process. There are often small
fixups (stylistic, minor nits, "I would have done it this way...", etc)
that are worth pointing out at the time, but are more painful to go back
and correct much later.
And documenting those discussions can really help other developers
besides the author and reviewer.
-Peff
^ permalink raw reply
* Re: diffcore-rename performance mode
From: David Kastrup @ 2007-09-25 19:32 UTC (permalink / raw)
To: git
In-Reply-To: <20070925190640.GA4613@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> On Tue, Sep 25, 2007 at 12:38:43PM -0400, Jeff King wrote:
>
>>[...]
>>
>> What is most confusing is why the 'somefree' case performs so badly,
>> since we should just be using the cnt_data. I'll see if gprof can shed
>
> OK, I found the problem. estimate_similarity calls
> diff_populate_filespec each time, even if we already have the cnt_data,
> which leads to recomputing the blob contents from deltas. Oops.
>
> Fixing this, the correct numbers are:
>
> | stock | nofree | old somefree | fixed somefree
> -----------------|-----------------------------------------------
> user time (s) | 76.78 | 16.96 | 46.26 | 16.99
> peak memory (Kb) | 52300 | 66796 | 59156 | 57328
>
> So now we're at a 4.5x speedup for about 10% extra memory usage. Patch
> will follow.
Sounds good except when we happen to just hit the "memory working set
exceeds physical memory" sweet spot. But the odds are much better
than for "nofree".
--
David Kastrup
^ permalink raw reply
* Re: Workflow question
From: Russ Brown @ 2007-09-25 19:42 UTC (permalink / raw)
To: git
In-Reply-To: <46F95CCC.4080209@op5.se>
Andreas Ericsson wrote:
> Russ Brown wrote:
>> Hi,
>>
>> I've been trying to think of a git workflow that we could use to replace
>> our current svn/svk setup without simply using git in exactly the same
>> way that we use svn/svk.
>>
>> Basically, we develop, maintain and enhance a website. On the central
>> repo is trunk which represents live, and any number of project branches.
>> Developers don't use local branches: they check out the project branches
>> they're working on and commit to those. Developers merge from trunk to
>> project branch from time to time to keep them current, and when a
>> project rolls out the branch is merged to trunk.
>>
>> In addition to the obvious advantages that git would give us (such as
>> properly tracking that code author as opposed to the person who did the
>> merge), I'm wanting to gain the following benefits:
>>
>> * The repository is very large (multiple gigabytes) and mirroring using
>> svk obviously takes a lot of time and space, so I'm keen to bring that
>> down, most likely by the developer not needing to mirror branches he
>> doesn't care about, or by being able to throw away branches he's done
>> with.
>> * The repository is full of revisions that fail review (or break
>> things) and are fixed by subsequent revisions. We'd much rather be able
>> to have the developer fix his revisions before they get committed
>> 'upstream' (whatever that ends up meaning).
>>
>> I asked earlier about the email-based model that git itself uses, and
>> while it appears to work very well for a widely-dispersed open-source
>> project, I think it will be too cumbersome and slow for a fast-paced
>> internal development team who make a number of live releases every day.
>>
>
> We came to the same conclusion at our workplace. Email works great, but
> it's faster and better to just walk over to your colleague and ask what
> he/she thinks about something.
>
Very true, with the minor exception that at my place there are
developers working at different sites, so the walk-over method will only
work for specific subsets of the team. :)
>> So, I've been thinking and have come up with this, which I'd appreciate
>> comments about:
>>
>> 1. On a server we stick a git repository which contains the master
>> branch, which represents what trunk did (i.e. the live platform). This
>> branch contains the full history for the live platform.
>
> A must-have for any more-than-two-developers setup, so so far so good ;-)
>
>> 2. On the same server we clone that repo to create a second repository
>> which is the developer area. In this we track master from the live repo,
>> and also create project branches.
>
> This isn't necessary. Branches in git are very nearly zero-cost, so having
> them in the same repo as the master branch won't hurt a bit. You can add
> an update-hook on the mothership repo to restrict access to the master
> branch if you like, but creating two separate repos will likely give
> more headache than it's worth.
>
Ah, right. I'm just trying to remember why it was I came up with that
idea in the first place, but I'm struggling a bit. :)
>> 3. Developers clone this developer repo, but I'd like them to be able
>> to decide which branches they actually want to clone from that
>> repository rather than simply cloning them all. Is this possible?
>
> Yes, although I'd actually recommend you to clone the full repo anyway.
> Since the various branches are likely to share quite a lot of history
> the added overhead of a few extra branches will most likely be negligible.
> git makes even very large codebases appear small and unobtrusive. The
> linux kernel history since 2.6.12 contains 554853 objects and compresses
> down to 178MiB.
>
Makes sense. Thing is, with git-svn (which I've been using for a while
now) it's possible to 'bring in' an upstream branch at will by adding it
to your config file, and on next git-svn fetch revisions that affect
that branch are automatically fetched. I figured it would just be a
little more efficient, though I appreciate it involves more 'fiddling'
by the client users.
Here's a question: is the creation and deletion of a branch also version
controlled as it is in Subversion? In other words, if I create a branch,
develop on it and delete it without merging it anywhere, will the
revisions hang around and get pulled down by future people cloning the
repository, or do they get thrown away?
> I think KDE is the largest repo imported to git so far. I've forgotten
> the exact numbers, but everyone was very impressed, and quite surprised,
> at the vast difference between SVN and git storage requirements.
>
>
>> 4. Developers create a local branch of the project they
>> are working on and commit to that.
>> 5. Once they think they're done, they publish their branch to the
>> development repo and request for comments.
>
> Using topic-branches is a much better strategy, usually, since that
> allows each feature to be evaluated and improved on on its own, rather
> than having to merge *all* of a particular developers changes just to
> get desirable feature X. Note that cherry-pick provides ways of doing
> that anyways, but in a much less elegant way, and your integrator/
> release engineer will likely tear his hair out on a daily basis without
> topic branches.
>
I've seen the term 'topic-branch' used here quite a bit but it's
unfamiliar to me. It is basically synonymous with what we call a
'project branch'? i.e. Management decide that feature X is required so
we create a branch to develop it on which all developers on the project
commit to.
Note that in step 4 above I mean the developer takes a local branch of
the topic branch. For example, we start projectX/main, and create branch
projectX on the shared repo. Developer 'jeff' works on the project and
so creates local branch projectX/jeff and begins work. In step 5 they
push this local branch to the shared repo so everyone can see it
(alternative to the 'walk-over' method or emailing). Note that all
changes jeff making in projectX/jeff are specific to the project branch,
so he can rebase against other changes that get committed to that
project branch.
If colleagues don't like his changes he can create projectXjeff2 and try
again.
Jeff can also have other local branches to keep separate changes he is
making on the other projects he is involved in.
That's how I'd thought of it happening...
>> 6. If all is not well, the developer creates a new local branch and
>> moves good revisions from his previous one to the new one, modifying
>> things as he goes, and republishes his new branch.
>> 7. If all is well, their works gets merged or rebased onto the main
>> project branch, and once that's ready it gets pushed to the master and
>> rolled to live. The developer's individual branches get deleted from the
>> dev repo since they're no longer required.
>
> Topic branches would work the same, basically, except they can be pushed up
> for review a lot faster.
>
> If all the pushing gets cumbersome, it also makes it easy to send the
> patches
> out as emails for discussion. It's usually easier to let git handle the
> actual code transmissions, but discussing patches in emails works quite
> well if it's intended for a wider audience.
>
Yes, I am going to experiment with this a little too to see just how
much work it would involve for the developers (if it's too much they
won't do it) :)
>> 8. From time to time the master branch gets merged to the project
>> branches. Developer's local branches can be rebased against the project
>> branch as they please.
>>
>
> criss-cross merging can turn kinda nasty though, as you may have a hard
> time
> finding *the* common point when you run into that rogue merge with conflict
> markers everywhere (it happens for everyone sooner or later).
>
> I'd suggest you rebase the developer/topic branches onto master with
> regular
> intervals instead.
>
Having been using git-svn for a while I really like the clean history
result that rebase gives, however my understanding was that you should
never rebase any published branch as it could screw up clones of that
branch. In fact, this is what has me the most confused: how to rebase a
project branch that is on a shared repository against master when
everyone will have it cloned? Or is this something that I clearly don't
understand properly?
(Thanks for your answers BTW Andreas)
--
Russ
^ permalink raw reply
* Re: [PATCH] Remove 'submodules' from the TODO section of the User Manual.
From: Miklos Vajna @ 2007-09-25 19:47 UTC (permalink / raw)
To: J. Bruce Fields; +Cc: Junio C Hamano, git
In-Reply-To: <20070925192315.GR30845@fieldses.org>
[-- Attachment #1: Type: text/plain, Size: 300 bytes --]
On Tue, Sep 25, 2007 at 03:23:15PM -0400, "J. Bruce Fields" <bfields@fieldses.org> wrote:
> Thanks, but it looks like Michael Smith already included this in his
> latest patch:
>
> http://marc.info/?l=git&m=119072428615724&w=2
ah, my bad. i missed that. sorry for the noise :)
- VMiklos
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [PATCH] Remove 'submodules' from the TODO section of the User Manual.
From: J. Bruce Fields @ 2007-09-25 19:49 UTC (permalink / raw)
To: Miklos Vajna; +Cc: Junio C Hamano, git
In-Reply-To: <20070925194745.GI18370@genesis.frugalware.org>
On Tue, Sep 25, 2007 at 09:47:45PM +0200, Miklos Vajna wrote:
> On Tue, Sep 25, 2007 at 03:23:15PM -0400, "J. Bruce Fields" <bfields@fieldses.org> wrote:
> > Thanks, but it looks like Michael Smith already included this in his
> > latest patch:
> >
> > http://marc.info/?l=git&m=119072428615724&w=2
>
> ah, my bad. i missed that. sorry for the noise :)
No problem.
And there's a ton of other stuff on that todo list if you're looking to
make more "noise"--patches always welcomed....
--b.
^ permalink raw reply
* Re: Workflow question
From: Wincent Colaiuta @ 2007-09-25 19:50 UTC (permalink / raw)
To: Jeff King; +Cc: Andreas Ericsson, Russ Brown, git
In-Reply-To: <20070925193416.GB8564@coredump.intra.peff.net>
El 25/9/2007, a las 21:34, Jeff King escribió:
> On Tue, Sep 25, 2007 at 09:09:00PM +0200, Andreas Ericsson wrote:
>
>> We came to the same conclusion at our workplace. Email works
>> great, but
>> it's faster and better to just walk over to your colleague and ask
>> what
>> he/she thinks about something.
>
> One of the projects I am working on does things this way, but I
> have to
> admit that I miss the email code-review process. There are often small
> fixups (stylistic, minor nits, "I would have done it this way...",
> etc)
> that are worth pointing out at the time, but are more painful to go
> back
> and correct much later.
>
> And documenting those discussions can really help other developers
> besides the author and reviewer.
Google has a pretty interesting internal code review system:
<http://video.google.com/videoplay?docid=-8502904076440714866>
Cheers,
Wincent
^ permalink raw reply
* Re: diffcore-rename performance mode
From: Jeff King @ 2007-09-25 19:52 UTC (permalink / raw)
To: David Kastrup; +Cc: git
In-Reply-To: <86641y4k34.fsf@lola.quinscape.zz>
On Tue, Sep 25, 2007 at 09:32:31PM +0200, David Kastrup wrote:
> > | stock | nofree | old somefree | fixed somefree
> > -----------------|-----------------------------------------------
> > user time (s) | 76.78 | 16.96 | 46.26 | 16.99
> > peak memory (Kb) | 52300 | 66796 | 59156 | 57328
> >
> > So now we're at a 4.5x speedup for about 10% extra memory usage. Patch
> > will follow.
>
> Sounds good except when we happen to just hit the "memory working set
> exceeds physical memory" sweet spot. But the odds are much better
> than for "nofree".
Of course, there is the possibility that it is the 10% that pushes you
into swap (or kills your ability to cache the entire pack in RAM).
However, this is probably not a big deal for two reasons:
1. this is a 50M process handling the linux-2.6 repository. Other
operations such as repacking already consume significantly more
memory (git-repack -a allocates 290M on the same repo).
2. I specifically turned off rename limiting (-l0) to do rename
detection on these large-ish diffs. If you have a machine which is
on the cusp of swapping, then don't do that. Jumping from -l100
(the default) to -l0 in my tests is _already_ moving you from 28M
to 52M, a much larger jump.
-Peff
^ permalink raw reply
* Re: [RFC] Convert builin-mailinfo.c to use The Better String Library.
From: David Kastrup @ 2007-09-25 19:55 UTC (permalink / raw)
To: git
In-Reply-To: <fdbmve$8lj$1@sea.gmane.org>
"Steven Burns" <royalstream@hotmail.com> writes:
> The C++ community in general suffers a lot from the NIH Syndrome.
> Matrixes, Strings, Vectors, everybody creates their own which are always, or
> course, superior to what's already available.
>
> Again, is not the language's fault, a language is just a language.
> It's the way it has been driven.
Having loose wires instead of a brake pedal in a car because the user
might prefer to brake with his teeth or by wiggling his backside or
building any other contraption of his own invention is a design
mistake. Especially when we are talking about public transportation
with changing drivers.
Making a language huge and bloated in order to be able to use the
language itself for defining a set of basic data types is just
masturbation. C++ has the most complicated set of implicit
conversions from any language in the world, and what for? It is
modeled for being able to create a user-defined "complex" type which
behaves almost as well as Fortran's. Too bad that this mostly means
everybody will define his own type (well, at least we have seen two or
three different library "standards" by now), and that the implicit
conversion rules and chains are appallingly wrong for a number of
other possible user-defined arithmetic types.
--
David Kastrup
^ permalink raw reply
* Re: [PATCH] Remove 'submodules' from the TODO section of the User Manual.
From: Miklos Vajna @ 2007-09-25 19:57 UTC (permalink / raw)
To: J. Bruce Fields; +Cc: Junio C Hamano, git
In-Reply-To: <20070925194936.GU30845@fieldses.org>
[-- Attachment #1: Type: text/plain, Size: 717 bytes --]
On Tue, Sep 25, 2007 at 03:49:36PM -0400, "J. Bruce Fields" <bfields@fieldses.org> wrote:
> And there's a ton of other stuff on that todo list if you're looking to
> make more "noise"--patches always welcomed....
i was this in the todo:
"Add a section on working with other version control systems, including
CVS, Subversion, and just imports of series of release tarballs."
to questions:
1) does this include documentation about git-svn? i mean a howto similar
to the submodules chapter.
2) does it worth to document a typical usage of tailor? we used it a lot
when converting from darcs to git and once you figure out the right
options, it's really easy to use it. but maybe it's unrelated.
thanks,
- VMiklos
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [PATCH] Remove 'submodules' from the TODO section of the User Manual.
From: J. Bruce Fields @ 2007-09-25 20:02 UTC (permalink / raw)
To: Miklos Vajna; +Cc: Junio C Hamano, git
In-Reply-To: <20070925195731.GJ18370@genesis.frugalware.org>
On Tue, Sep 25, 2007 at 09:57:31PM +0200, Miklos Vajna wrote:
> i was this in the todo:
>
> "Add a section on working with other version control systems, including
> CVS, Subversion, and just imports of series of release tarballs."
>
> to questions:
>
> 1) does this include documentation about git-svn? i mean a howto similar
> to the submodules chapter.
Could be. Though when I used it I thought the git-svn manpage was
actually very good, so I'd be tempted to just point to that.
> 2) does it worth to document a typical usage of tailor? we used it a lot
> when converting from darcs to git and once you figure out the right
> options, it's really easy to use it. but maybe it's unrelated.
I think we should try to keep the git documentation mostly confined to
tools that are distributed with git itself. So it might be worth a few
words just to mention the existance of tailor and how it compares to
other tools, but I'd leave it at that.
--b.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox