* Re: git-svnimport failed and now git-repack hates me
From: Linus Torvalds @ 2007-01-04 18:30 UTC (permalink / raw)
To: Chris Lee; +Cc: Git Mailing List
In-Reply-To: <204011cb0701040956p11ea2cepe3efaaf396056ac0@mail.gmail.com>
On Thu, 4 Jan 2007, Chris Lee wrote:
>
> Unfortunately, that's how the KDE repo is organized. (I tried arguing
> against this when they were going to do the original import, but I
> lost the argument.) And git-svnimport doesn't appear to have any sort
> of method for splitting a gigantic svn repo into several smaller git
> repos.
Well, the good news is, I think we could probably split it up from within
git. It's not fundamentally hard, although it is pretty damn expensive
(and it would require the subproject support to do really well).
So ignore that issue for now. I'd love to see the end result, if only
because it sounds like you have a test-case for git that is four times
bigger than the mozilla archive - even if it's just because of some really
really stupid design decisions from the KDE SVN maintainers ;)
(But I would actually expect that KDE SVN uses SVN subprojects, so
hopefully it's not _really_ one big repository. Of course, I don't know if
SVN really does subprojects or how well it does them, so that's just a
total guess).
The real problem with a SVN import is that I think SVN doesn't do merges
right, so you can't import merge history properly (well, you can, if you
decide that "properly" really means "SVN can't merge, so we can't really
show it as merges in git either").
I think both git-svn and git-svnimport can _guess_ about merges, but it's
just a heuristic, afaik. Whether it's a good one, I don't know.
> Yeah. I haven't bothered hacking git-svnimport yet - but it looks like
> having it automatically repack every thousand revisions or so would
> probably be a pretty big win.
That, or making it use the same "fastimport" that the hacked-up CVS
importer was made to use. Either way, somebody who understands SVN
intimately (and probably perl) would need to work on it.
That would not be me, so I can't really help ;)
> By default, if I had, say, one pack with the first 1000 revisions, and
> I imported another 1000, running 'git-repack' on its own would leave
> the first pack alone and create a new pack with just the second 1000
> revisions, right?
Yes. It's _probably_ better to do a full re-pack every once in a while
(because if you have a lot of pack-files, eventually that ends up being
problematic too), but as a first approximation, it's probably fine to just
do a plain "git repack" every thousand commits, and then do a full big
repack at the end.
The big repack will still be pretty expensive, but it should be less
painful than having everything unpacked. And at least the import won't
have run with millions and millions of loose objects.
So doing a "git repack -a -d" at the end is a good idea, and _maybe_ it
could be done in the middle too for really big packs.
Again, doing what fastimport does avoids most of the whole issue, since it
just generates a pack up-front instead. But that requires the importer to
specifically understand about that kind of setup.
> This is on a dual-CPU dual-core Opteron, running the AMD64 variant of
> Ubuntu's Edgy release (64-bit kernel, 64-bit native userland). The
> pack-file was around 2.3GB.
Ok, that should all be fine. A 31-bit thing in OpenSSL would explain it,
and doesn't sound unlikely. Just somebody using "int" somewhere, and it
would never have been triggered by any sane user of SHA1_Update(). The git
pack-check.c usage really _is_ very odd, even if it happens to make sense
in that particular schenario.
Linus
^ permalink raw reply
* Re: git-svnimport failed and now git-repack hates me
From: Chris Lee @ 2007-01-04 18:54 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0701041016010.3661@woody.osdl.org>
On 1/4/07, Linus Torvalds <torvalds@osdl.org> wrote:
> Well, the good news is, I think we could probably split it up from within
> git. It's not fundamentally hard, although it is pretty damn expensive
> (and it would require the subproject support to do really well).
I was hoping that'd be possible at some point. I really want to split
the submodules back out into first-class modules - one of my biggest
misgivings about the current KDE repository setup is how everything is
part of one gigantic repository.
> So ignore that issue for now. I'd love to see the end result, if only
> because it sounds like you have a test-case for git that is four times
> bigger than the mozilla archive - even if it's just because of some really
> really stupid design decisions from the KDE SVN maintainers ;)
The full on-disk size of the KDE SVN repo is about 37GB, last time I
checked. It may be up to 38 or 39GB now - I last ran rsync against the
svn repo a few weeks ago. I'm only focusing on importing the first
409k revisions at the moment, because that comprises the commits that
originally came from CVS and were imported into SVN. Almost
immediately after the CVS import, coolo made some changes - moving all
of the core KDE modules into /trunk/KDE, and their branches and tags
into /branches/KDE and /tags/KDE respectively. This, I suspect will
end up making things "fun" for the other part of the import, which is
another 200k revisions, give or take.
So, yes, I suspect it's quite a bit larger than Mozilla. I'm doing the
conversion to git as a test so that I can show some numbers to the KDE
guys; I'm not trying to campaign for a transition to git, but I think
it's definitely worth exploring what such a world would look like. But
in order for me to try to make a compelling argument for an eventual
project move to git, the git win32 support would need to be really
good. (In KDE4, we're supporting Windows and OS X as well as X11 as
first-class platforms.)
> (But I would actually expect that KDE SVN uses SVN subprojects, so
> hopefully it's not _really_ one big repository. Of course, I don't know if
> SVN really does subprojects or how well it does them, so that's just a
> total guess).
I don't think so, but I'll ask coolo (the KDE SVN administrator).
> The real problem with a SVN import is that I think SVN doesn't do merges
> right, so you can't import merge history properly (well, you can, if you
> decide that "properly" really means "SVN can't merge, so we can't really
> show it as merges in git either").
>
> I think both git-svn and git-svnimport can _guess_ about merges, but it's
> just a heuristic, afaik. Whether it's a good one, I don't know.
Not too worried about the merges right now - as long as I have a rough
approximation of what the original looked like, I'm pretty happy.
> > Yeah. I haven't bothered hacking git-svnimport yet - but it looks like
> > having it automatically repack every thousand revisions or so would
> > probably be a pretty big win.
>
> That, or making it use the same "fastimport" that the hacked-up CVS
> importer was made to use. Either way, somebody who understands SVN
> intimately (and probably perl) would need to work on it.
>
> That would not be me, so I can't really help ;)
Well, Shawn pointed me at the fastimport stuff, and I happen to know
Perl reasonably well (I think) so I'll take a stab at trying it that
way.
> > By default, if I had, say, one pack with the first 1000 revisions, and
> > I imported another 1000, running 'git-repack' on its own would leave
> > the first pack alone and create a new pack with just the second 1000
> > revisions, right?
>
> Yes. It's _probably_ better to do a full re-pack every once in a while
> (because if you have a lot of pack-files, eventually that ends up being
> problematic too), but as a first approximation, it's probably fine to just
> do a plain "git repack" every thousand commits, and then do a full big
> repack at the end.
Sounds like a good idea. Also sounds like it would be much less
painful than the current situation, where it takes over nine hours to
pack up all these revisions. :)
> The big repack will still be pretty expensive, but it should be less
> painful than having everything unpacked. And at least the import won't
> have run with millions and millions of loose objects.
>
> So doing a "git repack -a -d" at the end is a good idea, and _maybe_ it
> could be done in the middle too for really big packs.
Okay, good to know.
> Again, doing what fastimport does avoids most of the whole issue, since it
> just generates a pack up-front instead. But that requires the importer to
> specifically understand about that kind of setup.
I'll definitely be investigating the fastimport option. Looks like
I'll get to crack open some of my Perl books - haven't had to do that
in a while. :)
^ permalink raw reply
* [PATCH] Make check target depend on common-cmds.h
From: René Scharfe @ 2007-01-04 18:33 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
This fixes sparse complaining about a missing include file
if 'make check' is run on clean sources.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
Makefile | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index fa1a022..180e1e0 100644
--- a/Makefile
+++ b/Makefile
@@ -818,7 +818,7 @@ test-sha1$X: test-sha1.o $(GITLIBS)
check-sha1:: test-sha1$X
./test-sha1.sh
-check:
+check: common-cmds.h
for i in *.c; do sparse $(ALL_CFLAGS) $(SPARSE_FLAGS) $$i || exit; done
^ permalink raw reply related
* [PATCH] Remove shadowing variable from traverse_trees()
From: René Scharfe @ 2007-01-04 18:33 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
The variable named entry is allocated using malloc() and then
forgotten, it being shadowed by an automatic variable of the
same name. Fixing the array size at 3 worked so far because
the only caller of traverse_trees() needed only as much
entries. Simply remove the shadowing varaible and we're able
to traverse more than three trees and save stack space at the
same time!
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
tree-walk.c | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
diff --git a/tree-walk.c b/tree-walk.c
index 14cc5ae..22f4550 100644
--- a/tree-walk.c
+++ b/tree-walk.c
@@ -113,7 +113,6 @@ void traverse_trees(int n, struct tree_desc *t, const char *base, traverse_callb
struct name_entry *entry = xmalloc(n*sizeof(*entry));
for (;;) {
- struct name_entry entry[3];
unsigned long mask = 0;
int i, last;
^ permalink raw reply related
* Re: git-svnimport failed and now git-repack hates me
From: Chris Lee @ 2007-01-04 19:24 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0701031737300.4989@woody.osdl.org>
On 1/3/07, Linus Torvalds <torvalds@osdl.org> wrote:
> > Checking `top` and `ps` revealed that there were no git-svnimport
> > processes doing anything, but all of my 4G of RAM was still marked as
> > used by the kernel. I had to do sysctl -w vm.drop_caches=3 to get it
> > to free all the RAM that the svn import had used up.
>
> I think that was just all cached, and all ok. The reason you didn't see
> any git-svnimport was that it had died off already, and all your memory
> was just caches. You could just have left it alone, and the kernel would
> have started re-using the memory for other things even without any
> "drop_caches".
>
> But what you did there didn't make anything worse, it was just likely had
> no real impact.
Thought it was worth mentioning this:
When I checked top, the numbers it showed me were:
Mem: 4059332k total, 3216480k used, 842852k free, 40824k buffers
Swap: 0k total, 0k used, 0k free, 37364k cached
40MB in buffers, 37MB in cache, and 3GB used.
Seems like *something* was definitely lost there. The 'used' number
didn't go down at all when I started doing other things; it went up as
the new programs started, then they used up some RAM, and then when
they exited they'd free whatever resources they'd used. However, until
I did the drop_caches, that number stayed pretty damn big.
The system has been up since then, doing lots of things, and still
seems pretty stable, so I think it's okay, but I thought that it was
worth mentioning that something seemed to be leaky.
^ permalink raw reply
* clarification on git-update-index --remove
From: David Tweed @ 2007-01-04 19:29 UTC (permalink / raw)
To: git
Hi, can someone who understands the index in git please clarify something
for me? Looking at the documentation it's not totally clear which entites
are referred to by "removed" and what that then implies.
from git-1.4.4.3/Documentation/git-update-index.txt <<<<<<<<<<<
--remove::
If a specified file is in the index but is missing then it's
removed.
Default behavior is to ignore removed file.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
I currently take this to mean
"With --remove, if there is a filepath in the index
and that filepath is in the list of files being fed to this instance
of git-update-index but in the working directory tree that filepath
does not currently exist, then upon termination remove the
filepath's record from the index.
[This means at the next commit it will not be present
at all in the set of files recorded via this commit.]
Default behaviour in that situation is to leave the index's
current record unaltered.
[This means that that filename will be, upon a commit,
recorded as having whatever sha-content the index currently records
it as having.]"
If this reading is right, this means that I have to pass the names of
files that have been removed otherwise they'll stay with their "last
contents before removal"? I'm currently finding removed files
"stay around in snapshots with their ultimate contents", and I now
_think_ this is the reason but I'm not sure.
(In case anyone's wants to know the context, I traced through and
distilled git-commit.sh to what I think
I need (rexpressed in python) with the core "make snapshot" code being
<<<<<<<<<<<<<<<<
p1=loggedPopen(["git-update-index","-z","--add","--remove","--stdin"],
stdin=PIPE)
for (d,fs) in files.iteritems(): # enters only "to be recorded" items
for f in fs:
p1.stdin.write(os.path.join(d,f)+"\0")
p1.stdin.close()
p1.wait()
tree=loggedPopen(["git-write-tree"],stdout=PIPE).communicate()[0].rstrip()
commit=loggedPopen(["git-commit-tree",tree,"-p","HEAD"],
stdin=PIPE,stdout=PIPE).communicate("cv")[0].rstrip()
<<<<<<<<<<<<<<<<<
Here files contains the set of files to be recorded in the current snapshot,
structured as (directory,set of files in directory).)
Many thanks for any insight,
--
cheers, dave tweed__________________________
david "DOT" tweed "AT" gmail "DOT" com
Rm 124, School of Systems Engineering, University of Reading.
Details are all that matters; God dwells there, and you never get to
see Him if you don't struggle to get them right. -- Stephen Jay Gould
^ permalink raw reply
* Re: HTTP access via proxy
From: Guilhem Bonnefille @ 2007-01-04 19:53 UTC (permalink / raw)
To: git
In-Reply-To: <8b65902a0701030505s1a80ef62vb42b5fe1088218fb@mail.gmail.com>
I finally found something.
The bug is that git-http-fetch have succes with first HTTP GET, but
second loose authentification tokens. So, proxy reject access.
I tried with libcurl 7.16, and it works.
So, I think it is a libcurl problem.
On 1/3/07, Guilhem Bonnefille <guilhem.bonnefille@gmail.com> wrote:
> Hi,
>
> I'm new to GIT. I've just discovered it recenntly, I found it powerfull.
>
> Actually, I'm trying to "connect" to GIT repositories from office.
> There is a proxy. I set the http_proxy varenv. But the cloning failed.
>
> Seems to be a 407 error, but I do not understand what to do. Any help
> is welcome.
>
> Here is the output:
> $ git-clone http://repo.or.cz/r/cogito.git
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> Getting pack list for http://repo.or.cz/r/cogito.git/
> Getting index for pack 605b6b3b97cd6a8725f9be2c858812646de69340
> Getting index for pack fbfa016d2f36e7317c82ccae9ba84f98ee23c232
> Getting index for pack 3e83e7bd975592d35f507a83d9d72b8f6fa2f90c
> Getting index for pack 3718e548f38e28d2b5c0fa845ecde4fa8011097f
> Getting index for pack 2ca93e541574382d920bb2310d774e1c4ce3f097
> Getting index for pack ec503789c27cc660192d567defd4d4620e5ff433
> Getting index for pack be4d257c2efb91623aeb6031689f348aa897cfd6
> Getting index for pack b8d235df7313ca48648dedea9fd059d7fe0fc440
> Getting index for pack 05ee5f60265a8cf543e5a42c04af0f7dacdf7ce3
> Getting index for pack a33f5335e33bb274e7c838fc0dc6deec5713ca3f
> Getting index for pack 8530b481a7f02b3d09fe00c6b6afedc658933dc1
> Getting index for pack 4b7808cea29f690eaa10aacff9eac93a38b5ddd3
> Getting index for pack f8c8b06fec5010796ce83d3809d022e8e2ac1c16
> Getting index for pack 7ddc7c3ff0dc06aefe42ad477ba3be48a54b6c48
> Getting index for pack f3e337782d2bf95568a7bd96866c4bdffb2f183f
> Getting index for pack 298348359dbe6956c697261eb88b47d9e6c6666e
> Getting index for pack 57964c548d649be865d283d6ab3ca10bf302f082
> Getting index for pack 78202a8501eff4614d7bb1972902a6c880bb2002
> Getting index for pack 51412f3fc46ba6059457a5890369895a96a6b427
> Getting index for pack 9b452ef8de67af41587ebb196ca4518280ba1795
> Getting index for pack 56df4494fdb935bb2f8de3388c901be086cf9bd7
> Getting index for pack cd046245115aa0b088f1b53bba2e68e0000b1c54
> Getting index for pack bdf83dd134bd89eca9431f0d6b7601e32d516ae9
> Getting index for pack 48baac389492f0071f8713547371eaae5507868d
> Getting index for pack 8b4c56b9f2e0d9586b10c24ef87b933f8d3c0d72
> Getting index for pack 49e12d13de0cb60255b343c59eeaf61b168fa0de
> Getting index for pack 9af7de2760000428a9bb60ea5afc150bb8a60fb8
> Getting pack 9af7de2760000428a9bb60ea5afc150bb8a60fb8
> which contains b6a6e87cb3e1368ad0f78c18fdb6c29dde4ae83e
> walk b6a6e87cb3e1368ad0f78c18fdb6c29dde4ae83e
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> Getting pack 49e12d13de0cb60255b343c59eeaf61b168fa0de
> which contains 18d0bd75adae8c1994373e8e5afdb388268c048c
> error: Unable to get pack file
> http://repo.or.cz/r/cogito.git//objects/pack/pack-49e12d13de0cb60255b343c59eeaf61b168fa0de.pack
> The requested URL returned error: 407
> Getting alternates list for http://repo.or.cz/r/cogito.git/
> error: Unable to find 18d0bd75adae8c1994373e8e5afdb388268c048c under
> http://repo.or.cz/r/cogito.git/
> Cannot obtain needed commit 18d0bd75adae8c1994373e8e5afdb388268c048c
> while processing commit b6a6e87cb3e1368ad0f78c18fdb6c29dde4ae83e.
> Waiting for http://repo.or.cz/r/cogito.git/objects/c2/8d9943e5d8a51088266db92030480602c35fc1
>
>
> Surprisingly, if I download the pack file directly with curl, it works:
> curl http://repo.or.cz/r/cogito.git//objects/pack/pack-49e12d13de0cb60255b343c59eeaf61b168fa0de.pack
>
>
> I'm using:
> - a freshly downloaded GIT tarball.
> - curl-7.12.1-5.rhel4
> - curl-devel-7.12.1-5.rhel4
>
> --
> Guilhem BONNEFILLE
> -=- #UIN: 15146515 JID: guyou@im.apinc.org MSN: guilhem_bonnefille@hotmail.com
> -=- mailto:guilhem.bonnefille@gmail.com
> -=- http://nathguil.free.fr/
>
--
Guilhem BONNEFILLE
-=- #UIN: 15146515 JID: guyou@im.apinc.org MSN: guilhem_bonnefille@hotmail.com
-=- mailto:guilhem.bonnefille@gmail.com
-=- http://nathguil.free.fr/
^ permalink raw reply
* Re: [PATCH] Speedup recursive by flushing index only once for all entries
From: Junio C Hamano @ 2007-01-04 20:22 UTC (permalink / raw)
To: Alex Riesen; +Cc: Johannes Schindelin, Git Mailing List, Junio C Hamano
In-Reply-To: <81b0412b0701040447u329dcf9bvcd7adb9e9d199f18@mail.gmail.com>
"Alex Riesen" <raa.lkml@gmail.com> writes:
> Me too, just wondered why didn't we do this back then.
> Anyway, my "monster-merge" and the builtin tests pass with
> no visible problems.
>
>> However, I was wondering if the index has to be written at all.
>> I expect the written index (except the last one, of course) to have no
>> user...
>
> Good question...
That's most likely because you played safe, and started from the
Python version whose only way to manipulate the index and write
out a tree was to actually write the index out.
So let's step back a bit.
* The top-level interface is merge() function that takes two
heads and the ancestor, and is responsible for coming up with
a merged tree in *result if it can. The main() calls it to
see if things merge cleanly, and wants the index populated
with the final merge result, be it clean or unmerged.
* merge() in addition to two heads and their ancestor takes
call-depth because it does the "recursive" business. When it
operates with positive call-depth, it is coming up with a
virtual commit that has the merge result tree of two merge
bases. In this case index_only is set to true and what is in
the working tree does not matter (i.e. usual "your local
modification will be clobbered, cannot merge" check should
not apply [*1*]). It calls setup_index() to read in the true
index (for the outermost case) or a temporary index (from one
of the ancestor trees in the recursive case) before passing
control to the real workhorse, merge_trees().
What merge() does before its call to setup_index() does not
depend on what the index is (even in recursive case, because
the real work done by merge_trees() in the recursively called
merge() is preceded by the call to setup_index()). When
merge() returns, the index contents, both in-core and on the
filesystem, matter only for the outermost call to merge().
So that suggests that flush_cache() in main() needs to stay.
* merge_trees() takes the two heads and the ancestor, and comes
up with the merge result, using in-core index. It calls
git_merge_trees() which is 3-way "read-tree -m", and calls
git_write_tree() with two purposes: (1) to see if the merge
was cleanly done, and (2) to get the tree to be used as the
virtual ancestor (in other words, for the outermost merge,
writing of the tree is not important -- a tree is produced as
an unused side effect of seeing if the merge was clean). If
it results in a clean merge for the outermost merge, the call
to flush_cache() in main() is what matters -- the calling
script of merge-recursive writes a tree out from the index
written by that call. I'll address the need for
git_write_tree() to write the index shortly.
If the call to git_merge_trees() results in conflicted merge,
merge_trees() falls into the rename detection codepath. Most
of the work is done in process_renames(), which updates the
index, and then remaining unmatched paths are dealt with by
calling process_entry(), which also updates the index. These
index updates could all be done in-core without writing the
resulting index file out; we should not discard nor re-read
the index while they are processing one path at a time.
In a sense, merge_trees() does what 3-way "read-tree -m"
could have done if it were rename aware.
* git_merge_trees() is a bit questionable. It reads from the
current_index_file which is the true index for the outermost
merge or the temporary index populated from the 'head'
parameter give to it earlier by merge(). I think this use of
temporary index is not necessary. In other words, we could
start from an empty cache if index_only, I think.
And I think the reason git_write_tree() writes the index out
is because of this call to read_cache_from() at the beginning
of git_merge_trees(). It is told to write a tree out of the
current in-core index -- so I do not know why it needs to
call read_cache_from() to begin with.
Given the above analysis, it seems to me that the current code
too heavily inherited the invariant that the in-core and on
filesystem index should match at every step from the original
Python version. I think your patch goes in the right direction
to correct that, but it does not spell out the new invariant the
code assumes cleanly enough. For example, I do not think you
would want the call to flush_cache() in process_renames() for
the same reason you took out the call from process_entry() --
they both make many calls to update_file() and remove_file() to
touch the index and the working tree.
How about making the invariants to be:
upon entry of merge(), the in-core cache is populated as
appropriate for the merge. That is, it has the contents
of the true index for the outermost one, and discarded
for the virtual ancestor merge.
upon exit from merge(), the in-core cache holds the
merge result for that round of merge. That is, it is
suitable for flush_cache() to leave the final result for
the outermost merge, and it is a merged index that wrote
the virtual ancestor tree was written out from for inner
merges.
The codeflow would then become like this:
main() {
hold_lock_file_for_update(lock, git_path("index"), 1);
merge();
write_cache() || close || commit_lock_file();
}
merge() {
while (more than one ancestor) {
discard_cache();
merge(two ancestors using their common);
}
discard_cache();
if (call_depth == 0) {
read_cache();
index_only = 0;
}
else
index_only = 1;
merge_trees();
}
merge_trees() {
if (up to date)
return;
git_merge_trees();
if (in-core index is unmerged) {
process_renames();
}
if (index_only)
git_write_tree();
}
git_merge_trees() {
unpack_trees();
}
git_write_tree() {
if (stale cache tree)
cache_tree_update();
lookup_tree();
}
process_renames(), process_entry() {
call remove_file() and update_file() as needed,
trusting that the caller set up the in-core
index as appropriate and previous calls to these
functions left the in-core index to correctly
reflect what have been done so far.
}
By the way, Alex, you seem to heavily work on Cygwin, and I am
interested in your experience with Shawn's sliding mmap() on
real projects, as I suspect Cygwin would be more heavily
impacted with any mmap() related changes. You already said
performance is "bearable". Does that mean it was better and
got worse but still bearable, or it was unusably slow but now it
is bearably usable?
^ permalink raw reply
* Re: git-svnimport failed and now git-repack hates me
From: Junio C Hamano @ 2007-01-04 20:22 UTC (permalink / raw)
To: Chris Lee
Cc: Junio C Hamano, Linus Torvalds, Shawn Pearce, Sasha Khapyorsky,
Git Mailing List
In-Reply-To: <204011cb0701040958k884b613i8a4639201ae6443b@mail.gmail.com>
"Chris Lee" <chris133@gmail.com> writes:
> I'll try all of this after the pack is regenerated. Thanks!
Thank YOU for helping to make git better.
^ permalink raw reply
* Re: clarification on git-update-index --remove
From: Junio C Hamano @ 2007-01-04 20:27 UTC (permalink / raw)
To: David Tweed; +Cc: git
In-Reply-To: <e1dab3980701041129l3944df8ep985141d8f436d0a7@mail.gmail.com>
"David Tweed" <david.tweed@gmail.com> writes:
> --remove::
> If a specified file is in the index but is missing then it's
> removed.
> Default behavior is to ignore removed file.
If necessary, better rewording to remove confusion is very much
appreciated.
"update-index <path>..." without options (that is "Default
behaviour") updates the index using the contents of the named
<path> found on the filesystem. It does _not_ add new paths to
the index, nor remove paths (that exists in the index) from the
index.
Giving --add allows new paths to be added to the index if named
paths are found on the filesystem. Giving --remove allows
existing paths to be removed from the index if named paths are
not found on the filesystem.
^ permalink raw reply
* Adding spell checker to GIT
From: Deepak Barua @ 2007-01-04 20:46 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 314 bytes --]
Hi All,
I and sasikumar have designed and built a spell checker into
the pre-commit hook, could someone please test it and give us your
comments.
Just add the file with same name pre-commit hook to .git/hooks
directory in your git project and make it executable.
Regards
Deepak
--
Code Code Code Away
[-- Attachment #2: pre-commit --]
[-- Type: application/octet-stream, Size: 4184 bytes --]
#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by git-commit with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, make this file executable.
# This is slightly modified from Andrew Morton's Perfect Patch.
# Lines you introduce should not have trailing whitespace.
# Also check for an indentation that has SP before a TAB.
# Added Spell Checking Code to the existing code
# By Deepak Barua and Sasikumar Kandhasamy
if git-rev-parse --verify HEAD 2>/dev/null
then
git-diff-index -p -M --cached HEAD
else
# NEEDSWORK: we should produce a diff with an empty tree here
# if we want to do the same verification for the initial import.
:
fi |
perl -e '
# Adding the Aspell module to the perl script need aspell-dev installed in machine for the same
use Text::Aspell;
my $found_bad = 0;
my $filename;
my $reported_filename = "";
my $lineno;
my $continue = 0;
my $start_pattern = "";
my $end_pattern = "";
my $chosen_pattern = "";
my $match_condition = 0;
# Finding the file name extension
sub find_ext {
if($_[0] =~ /\.cc|\.java/) {
$start_pattern = qr!/\*|// !;
}
elsif($_[0] =~ /\.c|\.h/) {
$start_pattern = qr!/\*!;
}
else {
$start_pattern = qr!\# !;
$end_pattern = qq!\014!;
}
}
sub bad_line {
my ($why, $line) = @_;
if (!$found_bad) {
print STDERR "*\n";
print STDERR "* You have some suspicious patch lines:\n";
print STDERR "*\n";
$found_bad = 1;
}
if ($reported_filename ne $filename) {
print STDERR "* In $filename\n";
$reported_filename = $filename;
}
print STDERR "* $why (line $lineno)\n";
print STDERR "$filename:$lineno:$line\n";
}
# Created a spell checker instance
my $speller = Text::Aspell->new;
die unless $speller;
while (<>) {
if (m|^diff --git a/(.*) b/\1$|) {
$filename = $1;
find_ext($filename); # Calling the file extension finding function
next;
}
if (/^@@ -\S+ \+(\d+)/) {
$lineno = $1 - 1;
next;
}
if (/^ /) {
$lineno++;
next;
}
if (s/^\+//) {
$lineno++;
s/\n/ \014 /; # Substituting the newline in the file with junk octal character so chomp would not remove it
chomp;
# Main match condition for the start of the comment lines
if(($match_condition = m/($start_pattern)/) || $continue == 1) {
# Condition checking for not continuing comment and new comment pattern is matched
($continue != 1 && $match_condition == 1) ? $chosen_pattern = $+ : "Continue Pattern";
# Condition checking for not continuing comment and no new comment pattern is found
($continue != 1 && $match_condition != 1) ? $chosen_pattern = "" : "Continue Pattern";
# If chosen pattern is as specified then the corresponding end pattern is assigned
$chosen_pattern eq qq!/\*! ? $end_pattern=qq!\*/! : "End Pattern Not Found" ;
$chosen_pattern eq qq!//! ? $end_pattern=qq!\014! : "End Pattern Not Found" ;
# Split the line to individual words
@words = split / /,$_;
foreach $word (@words) {
# Check to see if end is reached
if($word eq $end_pattern) {
$continue=0;
$chosen_pattern="";
$end_pattern="";
last;
}
# Skip the start pattern
elsif($word eq $chosen_pattern) {
next;
}
# Perform spell checking for the selected word
else {
$speller->check($word)
? "Found"
: bad_line("\"$word\" spelling wrong"," ");
}
$continue = 1;
}
}
s/ \014 / /; # Substituting the junk octal with a space
# Continuation of the old code in pre-commit
if (/\s$/) {
bad_line("trailing whitespace", $_);
}
if (/^\s* /) {
bad_line("indent SP followed by a TAB", $_);
}
if (/^(?:[<>=]){7}/) {
bad_line("unresolved merge conflict", $_);
}
}
}
exit($found_bad);
'
^ permalink raw reply
* Re: [PATCH] gitweb: Fix "Use of uninitialized value" warning in git_tags_body
From: Junio C Hamano @ 2007-01-04 20:51 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <enh9h8$7l0$2@sea.gmane.org>
Jakub Narebski <jnareb@gmail.com> writes:
>> Fix "Use of uninitialized value" warning in git_tags_body generated
>> for lightweight tags of tree and blob object; those don't have age
>> ($tag{'age'}) defined.
>
> By the way, the latest "Fix warnings" patches for gitweb were result
> of working on bare bones test suite for gitweb. For now I'm only
> checking if there was anything written to STDERR.
I think this is a useful thing to have in-tree at this point,
with perhaps minor tweaks I'd suggest later. That way other
people can write more tests to catch regressions.
> BTW how in portable way to check that given file matches given patterns
> in specified order, or matches given patterns in any order?
This depends on how different the same page is rendered at
different times; I suspect the outputs for the same request from
two runs of the test are expected to be different in age, commit
and tag object ID and nothing else?
If so, I think using test_tick() to force the stable commit
object ID while your test is building a test repository would be
a useful thing to do. Also I think replacing call to "time"
(you seem to call it from multiple places, all to compute "age")
with a call to a subroutine "git_time" and then defining:
sub git_time {
if (exists $::ENV{GIT_TEST_TIME}) {
return $::ENV{GIT_TEST_TIME};
}
return time;
}
to help testing would be sensible. Set and export GIT_TEST_TIME
to a fixed time (say, midnight GMT December 28th, 2006) in
gitweb_init and you will get stable agestring if you have
already made the commit and author timestamps stable with
test_tick().
^ permalink raw reply
* Re: git-svnimport failed and now git-repack hates me
From: Linus Torvalds @ 2007-01-04 21:12 UTC (permalink / raw)
To: Chris Lee; +Cc: Git Mailing List
In-Reply-To: <204011cb0701041124g40440fd4udf1088ab1341c031@mail.gmail.com>
On Thu, 4 Jan 2007, Chris Lee wrote:
>
> Seems like *something* was definitely lost there. The 'used' number
> didn't go down at all when I started doing other things; it went up as
> the new programs started
The 'used' number basically _never_ goes down as long as there is memory
free. The kernel simply doesn't have any reason to free any of its caches,
even if those caches end up not being very useful.
What happened is almost certainly that with your big unpacked repository,
the kernel ended up using a lot of memory on filename caching. In other
words, I'd have expected that if you were to do
cat /proc/slabinfo
you'd have seen a _lot_ of memory being used for dentries ("dentry_cache")
and inodes ("ext3_inode_cache" assuming you're an ext3 user).
The kernel can easily drop those caches on demand, but "free" isn't quite
smart enough to know about them as being caches, so they will just show up
as "used".
That said, since you didn't want them, dropping them by hand with sysctl
certainly didn't hurt. Manual control can often be better than automatic
heuristics..
So the reason why repacking is so useful is that it gets rid of all these
millions of individual files. They all take up space on the disk, but they
also do end up having a lot of caches associated with them.
Btw, you may find that despite your 4GB of RAM, you might still be
better off with a swapfile. It gives the kernel a certain amount of
freedom in choosing how to allocate memory, and perhaps more importantly,
even when the kernel doesn't actively use it, it means that IF the kernel
runs out of totally free memory (because it has decided to keep a lot of
stuff in the dentry cache), it gives the kernel choices, and a certain
"buffer" for making the right decision.
What often happens is that the memory management heuristics don't make the
"perfect" choice (partly because it's theoretically impossible anyway, but
largely just because it's just a damn hard problem to even get all that
*close* to perfect), and having a swap partition or even a swap file just
allows the kernel to make some mistakes without it hitting a hard wall of
"oh, I can't do anything at all about this particular page".
So that buffer zone can be helpful in avoiding bad situations, but it can
actually also end up improving performance - it doesn't sound like the
case in this particular situation, but in some other loads there really
are a lot of dirty pages that aren't all that useful and where the memory
really could be better used for other things if the largely unused dirty
page could just be written to disk.
Linus
^ permalink raw reply
* Re: git-svnimport failed and now git-repack hates me
From: Sasha Khapyorsky @ 2007-01-04 21:31 UTC (permalink / raw)
To: Linus Torvalds, Chris Lee; +Cc: Junio C Hamano, Shawn Pearce, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0701031737300.4989@woody.osdl.org>
On 17:59 Wed 03 Jan , Linus Torvalds wrote:
>
> However, I don't know what the proper magic is for svnimport to do that
> sane "do it in chunks and tell when you're all done". Or even better - to
> just make it repack properly and not keep everything in memory.
> As to who knows how to fix git-svnimport to do something saner, I have no
> clue.. Sasha seems to have touched it last. Sasha?
I guess it should not be hard to do svnimport in incrementally with
repacking. Like this:
diff --git a/git-svnimport.perl b/git-svnimport.perl
index 071777b..afbbe63 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl
@@ -31,12 +31,13 @@ $SIG{'PIPE'}="IGNORE";
$ENV{'TZ'}="UTC";
our($opt_h,$opt_o,$opt_v,$opt_u,$opt_C,$opt_i,$opt_m,$opt_M,$opt_t,$opt_T,
- $opt_b,$opt_r,$opt_I,$opt_A,$opt_s,$opt_l,$opt_d,$opt_D,$opt_S,$opt_F,$opt_P);
+ $opt_b,$opt_r,$opt_I,$opt_A,$opt_s,$opt_l,$opt_d,$opt_D,$opt_S,$opt_F,
+ $opt_P,$opt_R);
sub usage() {
print STDERR <<END;
Usage: ${\basename $0} # fetch/update GIT from SVN
- [-o branch-for-HEAD] [-h] [-v] [-l max_rev]
+ [-o branch-for-HEAD] [-h] [-v] [-l max_rev] [-R repack_each_revs]
[-C GIT_repository] [-t tagname] [-T trunkname] [-b branchname]
[-d|-D] [-i] [-u] [-r] [-I ignorefilename] [-s start_chg]
[-m] [-M regex] [-A author_file] [-S] [-F] [-P project_name] [SVN_URL]
@@ -44,7 +45,7 @@ END
exit(1);
}
-getopts("A:b:C:dDFhiI:l:mM:o:rs:t:T:SP:uv") or usage();
+getopts("A:b:C:dDFhiI:l:mM:o:rs:t:T:SP:R:uv") or usage();
usage if $opt_h;
my $tag_name = $opt_t || "tags";
@@ -52,6 +53,7 @@ my $trunk_name = $opt_T || "trunk";
my $branch_name = $opt_b || "branches";
my $project_name = $opt_P || "";
$project_name = "/" . $project_name if ($project_name);
+my $repack_after = $opt_R || 1000;
@ARGV == 1 or @ARGV == 2 or usage();
@@ -938,11 +940,27 @@ if ($opt_l < $current_rev) {
exit;
}
-print "Fetching from $current_rev to $opt_l ...\n" if $opt_v;
+print "Processing from $current_rev to $opt_l ...\n" if $opt_v;
-my $pool=SVN::Pool->new;
-$svn->{'svn'}->get_log("/",$current_rev,$opt_l,0,1,1,\&commit_all,$pool);
-$pool->clear;
+my $from_rev;
+my $to_rev = $current_rev;
+
+while ($to_rev < $opt_l) {
+ $from_rev = $to_rev;
+ $to_rev = $from_rev + $repack_after;
+ $to_rev = $opt_l if $opt_l < $to_rev;
+ print "Fetching from $from_rev to $to_rev ...\n" if $opt_v;
+ my $pool=SVN::Pool->new;
+ $svn->{'svn'}->get_log("/",$from_rev,$to_rev,0,1,1,\&commit_all,$pool);
+ $pool->clear;
+ my $pid = fork();
+ die "Fork: $!\n" unless defined $pid;
+ unless($pid) {
+ exec("git-repack", "-d")
+ or die "Cannot repack: $!\n";
+ }
+ waitpid($pid, 0);
+}
unlink($git_index);
Chris, it works fine for me with small repository (~9000 revisions), but
I don't have such huge one as yours. Could you try? Thanks.
Sasha
^ permalink raw reply related
* Re: git-svnimport failed and now git-repack hates me
From: Chris Lee @ 2007-01-04 22:04 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Git Mailing List
In-Reply-To: <20070104213142.GE11861@sashak.voltaire.com>
> Chris, it works fine for me with small repository (~9000 revisions), but
> I don't have such huge one as yours. Could you try? Thanks.
Patch looks like it makes sense. I can definitely try it later.
Back to work for now...
^ permalink raw reply
* [BUG] stgit: unexpected "unknown user details" on refresh
From: Yann Dirson @ 2007-01-04 23:22 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
Hi Catalin,
Here is an error I just got when refreshing. The refresh did occur, but
the error is annoying and may hide something:
dwitch@gandelf:/export/work/yann/git/stgit$ ./stg ref contrib/stg-whatchanged
Checking for changes in the working directory... done
Refreshing patch "whatchanged"...stg ref: unknown user details
dwitch@gandelf:/export/work/yann/git/stgit$ echo $?
2
When using "refresh -p", we can see the error occurs before rewinding to
current patch:
dwitch@gandelf:/export/work/yann/git/stgit$ ./stg ref -p show-old
Checking for changes in the working directory... done
Popping patch "whatchanged"... done
Refreshing patch "show-old"...stg ref: unknown user details
dwitch@gandelf:/export/work/yann/git/stgit$ stg top
show-old
^ permalink raw reply
* [PATCH 0/5] My set of stgit companion scripts.
From: Yann Dirson @ 2007-01-04 23:39 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
The following series adds to contrib/ a set of bash scripts I find useful
for my daily use of stgit. They are probably closely tied to my workflow,
but others may find them useful as well. Some of them may be useful to
pythonize and integrate in some form or another.
stg-show-old is a trivial thing, but since I often need that functionnality,
it is much less comfortable than forging the same command-line over and over
again. It is also used by some of the other scripts.
stg-fold-files-from and stg-swallow are invaluable when shuffing changes
between patches.
stg-what-changed is invaluable when shuffling patches in the stack.
stg-cvs is a very young script, and probably very fragile. It allows a
lightweight way of preparing changes with stgit before committing them into
a CVS tree.
--
Yann Dirson <ydirson@altern.org> |
Debian-related: <dirson@debian.org> | Support Debian GNU/Linux:
| Freedom, Power, Stability, Gratis
http://ydirson.free.fr/ | Check <http://www.debian.org/>
^ permalink raw reply
* [PATCH 1/5] Add contrib/stg-show-old: show old version of a patch.
From: Yann Dirson @ 2007-01-04 23:46 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
In-Reply-To: <20070104233934.13580.17744.stgit@gandelf.nowhere.earth>
Signed-off-by: Yann Dirson <ydirson@altern.org>
---
contrib/stg-show-old | 13 +++++++++++++
1 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/contrib/stg-show-old b/contrib/stg-show-old
new file mode 100755
index 0000000..82692a7
--- /dev/null
+++ b/contrib/stg-show-old
@@ -0,0 +1,13 @@
+#!/bin/sh
+set -e
+
+# stg-show-old - mini helper to look at the previous version of a
+# patch (current one by default)
+
+# Copyright (c) 2006-2007 Yann Dirson <ydirson@altern.org>
+# Subject to the GNU GPL, version 2.
+
+[ "$#" -le 1 ] || { echo >&2 "Usage: $(basename $0) [<patch>]"; exit 1; }
+patch="$1"
+
+stg diff -r"${patch}//bottom.old".."${patch}//top.old"
^ permalink raw reply related
* [PATCH 2/5] Add contrib/stg-whatchanged: look at what would be changed by refreshing.
From: Yann Dirson @ 2007-01-04 23:46 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
In-Reply-To: <20070104233934.13580.17744.stgit@gandelf.nowhere.earth>
This script outputs a "metadiff" (diff of diffs, ie. diff between 2
versions of a single patch). This is somewhat a proof of concept, and some
work is indeed needed in the way to present the results. Consider
filtering the output at least through colordiff.
I have 2 uses for this script:
- when simply editing a patch, provides a 3rd way to check what I've done
before refreshing (in addition to "stg diff" and "stg diff -r //bottom")
- most usefully, when resolving conflicts caused by a push, to ease
checking of the merge result.
Signed-off-by: Yann Dirson <ydirson@altern.org>
---
contrib/stg-whatchanged | 37 +++++++++++++++++++++++++++++++++++++
1 files changed, 37 insertions(+), 0 deletions(-)
diff --git a/contrib/stg-whatchanged b/contrib/stg-whatchanged
new file mode 100755
index 0000000..978749c
--- /dev/null
+++ b/contrib/stg-whatchanged
@@ -0,0 +1,37 @@
+#!/bin/bash
+set -e
+
+# stg-whatchanged - show a metadiff for the patch being modified,
+# especially when resolving a merge.
+
+# Copyright (c) 2006-2007 Yann Dirson <ydirson@altern.org>
+# Subject to the GNU GPL, version 2.
+
+# FIXME:
+# - should only exclude hunk headers differing only in line offsets
+# - diff coloring should show changes in context lines differently than
+# changes in contents
+# - filter on ^index lines is a bit wide
+# - we should be able to ask diff to force a new hunk on "^@@ " to better
+# handle them
+# - we should always show the hunk header for any changes within a hunk
+
+# default to unified diff
+if [ "$#" = 0 ]; then
+ set -- -u
+fi
+
+# Merges via "push" leave top=bottom so we must look at old patch
+# in this case (unlike, eg., "pick --fold")
+patchdir="$(git-rev-parse --git-dir)/patches/$(stg branch)/$(stg top)"
+if [ $(cat "$patchdir/bottom") = $(cat "$patchdir/top") ];
+then
+ current_cmd="stg-show-old"
+else
+ current_cmd="stg show"
+fi
+
+colordiff "$@" \
+ -I '^index [0-9a-b]*..[0-9a-b]*' \
+ -I '^@@ .* @@' \
+ <($current_cmd) <(stg diff -r//bottom) | less -RFX
^ permalink raw reply related
* [PATCH 4/5] Add contrib/stg-swallow: completely merge an unapplied patch into current one
From: Yann Dirson @ 2007-01-04 23:46 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
In-Reply-To: <20070104233934.13580.17744.stgit@gandelf.nowhere.earth>
Signed-off-by: Yann Dirson <ydirson@altern.org>
---
contrib/stg-swallow | 19 +++++++++++++++++++
1 files changed, 19 insertions(+), 0 deletions(-)
diff --git a/contrib/stg-swallow b/contrib/stg-swallow
new file mode 100755
index 0000000..5014f39
--- /dev/null
+++ b/contrib/stg-swallow
@@ -0,0 +1,19 @@
+#!/bin/sh
+set -e
+
+# stg-swallow - completely merge an unapplied patch into current one
+
+# Copyright (c) 2006-2007 Yann Dirson <ydirson@altern.org>
+# Subject to the GNU GPL, version 2.
+
+# FIXME:
+# - should provide support for conflict solving ?
+
+[ "$#" = 1 ] || { echo >&2 "Usage: $(basename $0) <patch>"; exit 1; }
+patch="$1"
+
+stg pick --fold "$patch"
+stg refresh
+stg push "$patch"
+#stg clean "$patch"
+stg pop; stg clean -u
^ permalink raw reply related
* [PATCH 3/5] Add contrib/stg-fold-files-from: pick selected changes from a patch above current one
From: Yann Dirson @ 2007-01-04 23:46 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
In-Reply-To: <20070104233934.13580.17744.stgit@gandelf.nowhere.earth>
Signed-off-by: Yann Dirson <ydirson@altern.org>
---
contrib/stg-fold-files-from | 31 +++++++++++++++++++++++++++++++
1 files changed, 31 insertions(+), 0 deletions(-)
diff --git a/contrib/stg-fold-files-from b/contrib/stg-fold-files-from
new file mode 100755
index 0000000..bfb4a1f
--- /dev/null
+++ b/contrib/stg-fold-files-from
@@ -0,0 +1,31 @@
+#!/bin/sh
+set -e
+
+# stg-fold-files-from - picks changes to one file from another patch.
+# Only supports picking from one file, but allows to select any range
+# of hunks from the file, using the -# flag to filterdiff.
+# Use together with "filterdiff --annotate" in your diff pager, to
+# identify hunk numbers easily.
+
+# usage: stg-fold-files-from <patch> [-#<n>[-<n>][,<n>]...] <file>
+
+# Copyright (c) 2006-2007 Yann Dirson <ydirson@altern.org>
+# Subject to the GNU GPL, version 2.
+
+PATCH="$1"
+shift
+
+filtercmd=cat
+hunks=
+foldflags=
+while [ "$#" -gt 0 ]; do
+ case "$1" in
+ -\#*) hunks="$1"; shift ;;
+ -t) foldflags="-t"; shift ;;
+ -*) { echo >&2 "unknown flag '$1'"; exit 1; } ;;
+ *) break ;;
+ esac
+done
+[ "$#" = 1 ] || { echo >&2 "supports one file only"; exit 1; }
+
+stg show "$PATCH" | filterdiff -p1 $hunks -i "$1" | stg fold $foldflags
^ permalink raw reply related
* [PATCH 5/5] Add contrib/stg-cvs: helper script to manage a mixed cvs/stgit working copy
From: Yann Dirson @ 2007-01-04 23:47 UTC (permalink / raw)
To: Catalin Marinas; +Cc: GIT list
In-Reply-To: <20070104233934.13580.17744.stgit@gandelf.nowhere.earth>
This is an early prototype only, use with care, and be sure to read the
LIMITATIONS section in the script comments.
Signed-off-by: Yann Dirson <ydirson@altern.org>
---
contrib/stg-cvs | 126 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 126 insertions(+), 0 deletions(-)
diff --git a/contrib/stg-cvs b/contrib/stg-cvs
new file mode 100755
index 0000000..ee3e7fa
--- /dev/null
+++ b/contrib/stg-cvs
@@ -0,0 +1,126 @@
+#!/bin/bash
+set -e
+
+# stg-cvs - helper script to manage a mixed cvs/stgit working copy.
+
+# Allows quick synchronization of a cvs mirror branch (does not try to
+# reconstruct patchsets, creates "jumbo" commits), and commits stgit
+# patches to CVS.
+
+# Copyright (c) 2007 Yann Dirson <ydirson@altern.org>
+# Subject to the GNU GPL, version 2.
+
+# LIMITATIONS
+# - this is only a proof-of-concept prototype
+# - lacks an "init" command
+# - "commit" does not "cvs add/remove" any file or dir
+# - "commit" does not ensure the base is uptodate before trying to
+# commit (but hey, it's CVS ;)
+# - "commit" can only commit a single patch
+# - not much robustness here
+# - this only deals with CVS but could surely be extended to any other
+# VCS
+# - stg push/pop operations confuse cvsutils because of timestamp
+# changes
+# - lacks synchronisation of .cvsignore <-> .gitignore
+
+usage()
+{
+ [ "$#" = 0 ] || echo "ERROR: $*"
+ echo "Usage: $(basename $0) <command>"
+ echo " commands: $(do_commands)"
+ exit 1
+}
+
+do_commands()
+{
+ echo $(grep '^[a-z-]*)' $0 | cut -d')' -f1)
+}
+
+do_fetch()
+{
+ local return=0
+ local path
+
+ local parent="$1"
+ local branch="$2"
+
+ # record changes from cvs into index
+ stg branch "$parent"
+ cvs -fq update -dP | grep -v '^\? ' | tee /dev/tty | while read status path; do
+ if [ -e "$path" ]; then
+ git update-index "$path"
+ else
+ git rm "$path"
+ fi
+ done
+
+ # create commit
+ if git commit -m "stg-cvs sync"; then
+ :
+ else
+ return=$?
+ fi
+
+ # back to branch
+ stg branch "$branch"
+
+ return $return
+}
+
+# get context
+branch=$(stg branch)
+parent=$(git-repo-config "stgit.yd.${branch}.parent") ||
+ usage "no declared parent for '$branch' - set stgit.yd.${branch}.parent"
+
+# extract command
+
+[ "$#" -ge 1 ] || usage
+command="$1"
+shift
+
+case "$command" in
+fetch)
+ do_fetch "$parent" "$branch"
+ ;;
+
+pull)
+ if do_fetch "$parent" "$branch"; then
+ # update
+ stg pull --merged . "$parent"
+ stg clean --applied
+ fi
+ ;;
+
+commit)
+ # sanity asserts
+ [ $(stg applied | wc -l) = 1 ] ||
+ usage "you don't have exactly one patch applied"
+
+ # context
+ patch=$(stg top)
+
+ # commit
+ cvs -fq commit \
+ -F ".git/patches/$branch/patches/$patch/description" \
+ $(stg files --bare)
+
+ # sync the parent branch
+ stg branch "$parent"
+ git-cherry-pick "patches/${branch}/${patch}"
+ stg branch "${branch}"
+
+ # update
+ stg pull --merged . "$parent"
+ stg clean --applied
+ ;;
+
+_commands)
+ # hint for bash-completion people :)
+ do_commands
+ ;;
+
+*)
+ usage "unknown command '$command'"
+ ;;
+esac
^ permalink raw reply related
* Bugs in gitweb
From: Luben Tuikov @ 2007-01-05 1:29 UTC (permalink / raw)
To: git
gitweb has two bugs in git_patchset_body (now you know who
you are without data-mining with git-blame).
The first bug is a missing "</div>" on commitdiff.
Saw this one a couple of days ago, but was swamped
and had no time to post.
The second bug is displaying a renamed file with the
same name as the one of the preceding "patch". Saw
this one today when I pulled.
To exhibit these, you need to "commitdiff" a commit
which has at least one rename and at least one regular
diff preceding the rename.
Can the guilty parties (you know who you are) please
fix this.
"And while at it" can you please actually *do* "refactor"
git_patchset_body *into smaller functions each one doing
a single particular task*.
It is sad to see git_patchset_body in such despicable state
all the while seeing words like "refactor" in the commit logs
of that function.
git_patchset_body is grossly overloaded for what it is
supposed to do to, and being one single huge blob, it is
hard to maintain.
Thanks,
Luben
^ permalink raw reply
* [1/2 PATCH] git-svn: make multi-init less confusing
From: Eric Wong @ 2007-01-05 2:02 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, David Kågedal
In-Reply-To: <87fyaqvgoz.fsf@morpheus.local>
It now requires at least one of the (trunk|branch|tags) arguments
(either from the command-line or in .git/config). Also we make
sure that anything that is passed as a URL ('help') in David's
case is actually a URL.
Thanks to David Kågedal for reporting this issue.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-svn.perl | 78 +++++++++++++++++++++++++++++----------------------------
1 files changed, 40 insertions(+), 38 deletions(-)
diff --git a/git-svn.perl b/git-svn.perl
index b28c5bb..0fc386a 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -571,28 +571,25 @@ sub graft_branches {
sub multi_init {
my $url = shift;
- $_trunk ||= 'trunk';
- $_trunk =~ s#/+$##;
- $url =~ s#/+$## if $url;
- if ($_trunk !~ m#^[a-z\+]+://#) {
- $_trunk = '/' . $_trunk if ($_trunk !~ m#^/#);
- unless ($url) {
- print STDERR "E: '$_trunk' is not a complete URL ",
- "and a separate URL is not specified\n";
- exit 1;
- }
- $_trunk = $url . $_trunk;
- }
- my $ch_id;
- if ($GIT_SVN eq 'git-svn') {
- $ch_id = 1;
- $GIT_SVN = $ENV{GIT_SVN_ID} = 'trunk';
+ unless (defined $_trunk || defined $_branches || defined $_tags) {
+ usage(1);
}
- init_vars();
- unless (-d $GIT_SVN_DIR) {
- print "GIT_SVN_ID set to 'trunk' for $_trunk\n" if $ch_id;
- init($_trunk);
- command_noisy('repo-config', 'svn.trunk', $_trunk);
+ if (defined $_trunk) {
+ my $trunk_url = complete_svn_url($url, $_trunk);
+ my $ch_id;
+ if ($GIT_SVN eq 'git-svn') {
+ $ch_id = 1;
+ $GIT_SVN = $ENV{GIT_SVN_ID} = 'trunk';
+ }
+ init_vars();
+ unless (-d $GIT_SVN_DIR) {
+ if ($ch_id) {
+ print "GIT_SVN_ID set to 'trunk' for ",
+ "$trunk_url ($_trunk)\n";
+ }
+ init($trunk_url);
+ command_noisy('repo-config', 'svn.trunk', $trunk_url);
+ }
}
complete_url_ls_init($url, $_branches, '--branches/-b', '');
complete_url_ls_init($url, $_tags, '--tags/-t', 'tags/');
@@ -872,29 +869,34 @@ sub rec_fetch {
}
}
+sub complete_svn_url {
+ my ($url, $path) = @_;
+ $path =~ s#/+$##;
+ $url =~ s#/+$## if $url;
+ if ($path !~ m#^[a-z\+]+://#) {
+ $path = '/' . $path if ($path !~ m#^/#);
+ if (!defined $url || $url !~ m#^[a-z\+]+://#) {
+ fatal("E: '$path' is not a complete URL ",
+ "and a separate URL is not specified\n");
+ }
+ $path = $url . $path;
+ }
+ return $path;
+}
+
sub complete_url_ls_init {
- my ($url, $var, $switch, $pfx) = @_;
- unless ($var) {
+ my ($url, $path, $switch, $pfx) = @_;
+ unless ($path) {
print STDERR "W: $switch not specified\n";
return;
}
- $var =~ s#/+$##;
- if ($var !~ m#^[a-z\+]+://#) {
- $var = '/' . $var if ($var !~ m#^/#);
- unless ($url) {
- print STDERR "E: '$var' is not a complete URL ",
- "and a separate URL is not specified\n";
- exit 1;
- }
- $var = $url . $var;
- }
- my @ls = libsvn_ls_fullurl($var);
- my $old = $GIT_SVN;
+ my $full_url = complete_svn_url($url, $path);
+ my @ls = libsvn_ls_fullurl($full_url);
defined(my $pid = fork) or croak $!;
if (!$pid) {
- foreach my $u (map { "$var/$_" } (grep m!/$!, @ls)) {
+ foreach my $u (map { "$full_url/$_" } (grep m!/$!, @ls)) {
$u =~ s#/+$##;
- if ($u !~ m!\Q$var\E/(.+)$!) {
+ if ($u !~ m!\Q$full_url\E/(.+)$!) {
print STDERR "W: Unrecognized URL: $u\n";
die "This should never happen\n";
}
@@ -912,7 +914,7 @@ sub complete_url_ls_init {
waitpid $pid, 0;
croak $? if $?;
my ($n) = ($switch =~ /^--(\w+)/);
- command_noisy('repo-config', "svn.$n", $var);
+ command_noisy('repo-config', "svn.$n", $full_url);
}
sub common_prefix {
--
Eric Wong
^ permalink raw reply related
* [PATCH 2/2] git-svn: update documentation for multi-{init|fetch}
From: Eric Wong @ 2007-01-05 2:04 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, David Kågedal
In-Reply-To: <87fyaqvgoz.fsf@morpheus.local>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
Documentation/git-svn.txt | 54 ++++++++++++++++++++++++++++++--------------
1 files changed, 37 insertions(+), 17 deletions(-)
diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt
index f5f57e8..f754d2f 100644
--- a/Documentation/git-svn.txt
+++ b/Documentation/git-svn.txt
@@ -3,7 +3,7 @@ git-svn(1)
NAME
----
-git-svn - bidirectional operation between a single Subversion branch and git
+git-svn - bidirectional operation between Subversion and git
SYNOPSIS
--------
@@ -11,24 +11,20 @@ SYNOPSIS
DESCRIPTION
-----------
-git-svn is a simple conduit for changesets between a single Subversion
-branch and git. It is not to be confused with gitlink:git-svnimport[1].
-They were designed with very different goals in mind.
+git-svn is a simple conduit for changesets between Subversion and git.
+It is not to be confused with gitlink:git-svnimport[1], which is
+read-only and geared towards tracking multiple branches.
-git-svn is designed for an individual developer who wants a
+git-svn was originally designed for an individual developer who wants a
bidirectional flow of changesets between a single branch in Subversion
-and an arbitrary number of branches in git. git-svnimport is designed
-for read-only operation on repositories that match a particular layout
-(albeit the recommended one by SVN developers).
+and an arbitrary number of branches in git. Since its inception,
+git-svn has gained the ability to track multiple branches in a manner
+similar to git-svnimport; but it cannot (yet) automatically detect new
+branches and tags like git-svnimport does.
-For importing svn, git-svnimport is potentially more powerful when
-operating on repositories organized under the recommended
-trunk/branch/tags structure, and should be faster, too.
-
-git-svn mostly ignores the very limited view of branching that
-Subversion has. This allows git-svn to be much easier to use,
-especially on repositories that are not organized in a manner that
-git-svnimport is designed for.
+git-svn is especially useful when it comes to tracking repositories
+not organized in the way Subversion developers recommend (trunk,
+branches, tags directories).
COMMANDS
--------
@@ -370,7 +366,7 @@ SVN was very wrong.
Basic Examples
~~~~~~~~~~~~~~
-Tracking and contributing to a Subversion-managed project:
+Tracking and contributing to a the trunk of a Subversion-managed project:
------------------------------------------------------------------------
# Initialize a repo (like git init-db):
@@ -388,6 +384,30 @@ Tracking and contributing to a Subversion-managed project:
git-svn show-ignore >> .git/info/exclude
------------------------------------------------------------------------
+Tracking and contributing to an entire Subversion-managed project
+(complete with a trunk, tags and branches):
+See also:
+'<<tracking-multiple-repos,Tracking Multiple Repositories or Branches>>'
+
+------------------------------------------------------------------------
+# Initialize a repo (like git init-db):
+ git-svn multi-init http://svn.foo.org/project \
+ -T trunk -b branches -t tags
+# Fetch remote revisions:
+ git-svn multi-fetch
+# Create your own branch of trunk to hack on:
+ git checkout -b my-trunk remotes/trunk
+# Do some work, and then commit your new changes to SVN, as well as
+# automatically updating your working HEAD:
+ git-svn dcommit -i trunk
+# Something has been committed to trunk, rebase the latest into your branch:
+ git-svn multi-fetch && git rebase remotes/trunk
+# Append svn:ignore settings of trunk to the default git exclude file:
+ git-svn show-ignore -i trunk >> .git/info/exclude
+# Check for new branches and tags (no arguments are needed):
+ git-svn multi-init
+------------------------------------------------------------------------
+
REBASE VS. PULL
---------------
--
1.5.0.rc0.g0d67
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox