* Re: cvsimport fails with cvsps core dump
From: Rajkumar S @ 2006-09-14 7:22 UTC (permalink / raw)
To: git
In-Reply-To: <64de5c8b0609120914p4ea47860x60a294093d43bc24@mail.gmail.com>
On 9/12/06, Rajkumar S <rajkumars+git@gmail.com> wrote:
> While trying to cvsimport pfsense [1] source [2], cvsps coredumps and
> cvsimport aborts with the following error.
the function read_line does not check if the string is bigger that the
space alloted, and core dumps when such a string is encountered.
I have a small patch which prevents the crashing, by stopping the copy
when the buffer is full.
diff --git a/cvs_direct.c b/cvs_direct.c
index 920487d..3857ce2 100644
--- a/cvs_direct.c
+++ b/cvs_direct.c
@@ -679,8 +679,11 @@ static int read_line(CvsServerCtx * ctx,
}
p++;
len++;
+ if (len >= BUFSIZ){
+ *p = 0;
+ break;
+ }
}
-
return len;
}
raj
^ permalink raw reply related
* Re: [PATCH] Trivial support for cloning and fetching via ftp://.
From: Junio C Hamano @ 2006-09-14 6:57 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: git
In-Reply-To: <20060914022404.GA900@sashak.voltaire.com>
Sasha Khapyorsky <sashak@voltaire.com> writes:
> This adds trivial support for cloning and fetching via ftp://.
Interesting.
I was wondering myself if our use of curl libraries in
http-fetch allows us to do this when I was looking at the
alternates breakage yesterday.
At a few places we do look at http error code that is returned
from the curl library, and change our behaviour based on that.
But it appears the difference between error code from ftp and
http has no bad effect on us. In an empty repository, we can
run this:
$ git-http-fetch -a -v heads/merge \
ftp://ftp.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc.git
(of course, this should normally be with http://www.kernel.org).
We notice that we get an error from a request for one object,
and switch to pack & alternates transfer. The only difference
between http://www and ftp://ftp is that for the former we know
error code 404 and supress the error message but for the latter
we do not treat error 550 from RETR response any specially and
show an error message. We still fall back to retrieve packs,
hoping that the missing object is in a pack.
I'd take this patch as is, but we might want to add some error
message supression logic just like we do for http.
^ permalink raw reply
* [PATCH] gitweb: Use File::Find::find in git_get_projects_list
From: Jakub Narebski @ 2006-09-14 6:39 UTC (permalink / raw)
To: git
Earlier code to get list of projects when $projects_list is a
directory (e.g. when it is equal to $projectroot) had a hardcoded flat
(one level) list of directories. Allow for projects to be in
subdirectories also for $projects_list being a directory by using
File::Find.
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This doesn't add much overhead to "project_list" view,
compared to previous version; times are the same within margin
of error.
gitweb/gitweb.perl | 29 +++++++++++++++++++++--------
1 files changed, 21 insertions(+), 8 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index c3544dd..470bff2 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -715,16 +715,29 @@ sub git_get_projects_list {
if (-d $projects_list) {
# search in directory
my $dir = $projects_list;
- opendir my ($dh), $dir or return undef;
- while (my $dir = readdir($dh)) {
- if (-e "$projectroot/$dir/HEAD") {
- my $pr = {
- path => $dir,
- };
- push @list, $pr
+ my $pfxlen = length("$dir");
+
+ sub wanted {
+ # skip dot files (hidden files), check only directories
+ #return if (/^\./);
+ return unless (-d $_);
+
+ my $subdir = substr($File::Find::name, $pfxlen + 1);
+ # we check related file in $projectroot
+ if (-e "$projectroot/$subdir/HEAD") {
+ push @list, { path => $subdir };
+ $File::Find::prune = 1;
}
}
- closedir($dh);
+
+ File::Find::find({
+ no_chdir => 1, # do not change directory
+ follow_fast => 1, # follow symbolic links
+ #follow_skip => 2, # ignore duplicated files and directories
+ dangling_symlinks => 0, # ignore dangling symlinks, silently
+ wanted => \&wanted,
+ }, "$dir");
+
} elsif (-f $projects_list) {
# read from file(url-encoded):
# 'git%2Fgit.git Linus+Torvalds'
--
1.4.2
^ permalink raw reply related
* Re: [PATCH 2/2] Handle invalid argc gently
From: Junio C Hamano @ 2006-09-14 6:34 UTC (permalink / raw)
To: Dmitry V. Levin; +Cc: git
In-Reply-To: <20060914013146.GD7044@basalt.office.altlinux.org>
"Dmitry V. Levin" <ldv@altlinux.org> writes:
> On Wed, Sep 13, 2006 at 06:12:02PM -0700, Junio C Hamano wrote:
>> "Dmitry V. Levin" <ldv@altlinux.org> writes:
>>
>> > describe, git: Handle argc==0 case the same way as argc==1.
>>
>> When does one get (ac == 0)?
>
> When one executes something like
> execlp("git", NULL, NULL);
Fair enough.
What is the valid reason to do execlp("git", NULL, NULL)?
^ permalink raw reply
* Re: cvs import
From: Michael Haggerty @ 2006-09-14 5:36 UTC (permalink / raw)
To: Jon Smirl
Cc: Markus Schiltknecht, Martin Langhoff, Git Mailing List,
monotone-devel, dev
In-Reply-To: <9e4733910609131438n686b6d72u4d5799533c7473d7@mail.gmail.com>
Jon Smirl wrote:
> On 9/13/06, Markus Schiltknecht <markus@bluegap.ch> wrote:
>> Martin Langhoff wrote:
>> > On 9/14/06, Jon Smirl <jonsmirl@gmail.com> wrote:
>> >> Let's copy the git list too and maybe we can come up with one importer
>> >> for everyone.
That would be great.
> AFAIK none of the CVS converters are using the dependency algorithm.
> So the proposal on the table is to develop a new converter that uses
> the dependency data from CVS to form the change sets and then outputs
> this data in a form that all of the backends can consume. Of course
> each of the backends is going to have to write some code in order to
> consume this new import format.
Frankly, I think people are getting the priorities wrong by focusing on
the format of the output of cvs2svn. Hacking a new output format onto
cvs2svn is a trivial matter of a couple hours of programming.
The real strength of cvs2svn (and I can say this without bragging
because most of this was done before I got involved in the project) is
that it handles dozens of peculiar corner cases and bizarre CVS
perversions, including a good test suite containing lots of twisted
little example repositories. This is 90% of the intellectual content of
cvs2svn.
I've spent many, many hours refactoring and reengineering cvs2svn to
make it easy to modify and add new features. The main thing that I want
to change is to use the dependency graph (rather than timestamps tweaked
to reflect dependency ordering) to deduce changesets. But I would never
think of throwing away the "old" cvs2svn and starting anew, because then
I would have to add all the little corner cases again from scratch.
It would be nice to have a universal dumpfile format, but IMO not
critical. The only difference between our SCMs that might be difficult
to paper over in a universal dumpfile is that SVN wants its changesets
in chronological order, whereas I gather that others would prefer the
data in dependency order branch by branch.
I say let cvs2svn (or if you like, we can rename it to "cvs2noncvs" :-)
) reconstruct the repository's change sets, then let us build several
backends that output the data in the format that is most convenient for
each project.
Michael
^ permalink raw reply
* Re: cvs import
From: Michael Haggerty @ 2006-09-14 5:35 UTC (permalink / raw)
To: Martin Langhoff
Cc: Jon Smirl, Markus Schiltknecht, Git Mailing List, monotone-devel,
dev
In-Reply-To: <46a038f90609132221o125c4694r75dbc8f728104832@mail.gmail.com>
Martin Langhoff wrote:
> On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
>> 2. Long-term continuous mirroring (backwards and forwards) between CVS
>> and another SCM, to allow people to use their preferred tool. (I
>> actually think that this is a silly idea, but some people seem to like
>> it.)
>
> Call me silly ;-) I use this all the time to track projects that use
> CVS or SVN, where I either
>
> [...]
Sorry, I guess I was speaking as a person who prefers and is most
familiar with centralized SCM. But I see from your response that the
ultimate in decentralized development is that each developer decides
what SCM to use :-) and that incremental conversion makes sense in that
context.
Michael
^ permalink raw reply
* Re: cvs import
From: Jon Smirl @ 2006-09-14 5:30 UTC (permalink / raw)
To: Michael Haggerty
Cc: Martin Langhoff, Markus Schiltknecht, Git Mailing List,
monotone-devel, dev
In-Reply-To: <4508E26B.5000106@alum.mit.edu>
On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> Jon Smirl wrote:
> > On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> >> But aside from this point, I think an intrinsic part of implementing
> >> incremental conversion is "convert the subsequent changes to the CVS
> >> repository *subject to the constraints* imposed by decisions made in
> >> earlier conversion runs. And the real trick is that things can be done
> >> in CVS (e.g., line-end changes, manual copying of files in the repo)
> >> that (a) are unversioned and (b) have retroactive effects that go
> >> arbitrarily far back in time. This is the reason that I am pessimistic
> >> that incremental conversion will ever work robustly.
> >
> > We don't need really robust incremental conversion. It just needs to
> > work most of the time. Incremental conversion is usually used to track
> > the main CVS repo with the new tool while people decide if they like
> > the new tool. Commits will still flow to the CVS repo and get
> > incrementally copied to the new tool so that it tracks CVS in close to
> > real time.
>
> I hadn't thought of the idea of using incremental conversion as an
> advertising method for switching SCM systems :-) But if changes flow
> back to CVS, doesn't this have to be pretty robust?
Changes flow back to CVS but using the new tool to generate a patch,
apply the patch to your CVS check out and commit it.
There are too many people working on Mozilla to get agreement to
switch in a short amount of time. git may need to mirror CVS for
several months. There are also other people pushing svn, monotone,
perforce, etc, etc, etc. Bottom line, Mozilla really needs a
distributed system because external companies are making large changes
and want their repos in house.
In my experience none of the other SCMs are up to taking one Mozilla
yet. Git has the tools but I can get a clean import.
I am using this process on Mozilla right now with git. I have a script
that updates my CVS tree overnight and then commits the changes into a
local git repo. I can then work on Mozilla using git but my history is
all messed up. When a change is ready I generate a diff against last
night's check out and apply it to my CVS tree and commit. CVS then
finds any merge problems for me.
>
> In our trial period, we simply did a single conversion to SVN and let
> people play with this test repository. When we decided to switch over
> we did another full conversion and simply discarded the changes that had
> been made in the test SVN repository.
>
> The use cases that I had considered were:
>
> 1. For conversions that take days, one could do a full commit while
> leaving CVS online, then take CVS offline and do only an incremental
> conversion to reduce SCM downtime. This is of course less of an issue
> if you could bring the conversion time down to a couple hours for even
> the largest CVS repos.
>
> 2. Long-term continuous mirroring (backwards and forwards) between CVS
> and another SCM, to allow people to use their preferred tool. (I
> actually think that this is a silly idea, but some people seem to like it.)
>
> For both of these applications, incremental conversion would have to be
> robust (for 1 it would at least have to give a clear indication of
> unrecoverable errors).
>
>
> Michael
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: cvs import
From: Martin Langhoff @ 2006-09-14 5:21 UTC (permalink / raw)
To: Michael Haggerty
Cc: Jon Smirl, Markus Schiltknecht, Git Mailing List, monotone-devel,
dev
In-Reply-To: <4508E26B.5000106@alum.mit.edu>
On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> 2. Long-term continuous mirroring (backwards and forwards) between CVS
> and another SCM, to allow people to use their preferred tool. (I
> actually think that this is a silly idea, but some people seem to like it.)
Call me silly ;-) I use this all the time to track projects that use
CVS or SVN, where I either
- Do have write access, but often develop offline (and I have a bunch
of perl/shell scripts to extract the patches and auto-commit them into
CVS/SVN).
- Do have write access, but want to experimental work branches
without making much noise in the cvs repo -- and being able to merge
CVS's HEAD in repeatedly as you'd want.
- Run "vendor-branch-tracking" setups for projects where I have a
custom branch of a FOSS sofware project, and repeatedly import updates
from upstream. this is the 'killer-app' of DSCMs IMHO.
It is not as robust as I'd like; with CVS, the git imports eventually
stray a bit from upstream, and requires manual fixing. But it is
_good_.
cheers,
martin
^ permalink raw reply
* Re: cvs import
From: Michael Haggerty @ 2006-09-14 5:02 UTC (permalink / raw)
To: Jon Smirl
Cc: Martin Langhoff, Markus Schiltknecht, Git Mailing List,
monotone-devel, dev
In-Reply-To: <9e4733910609132134j63857912keed6a42682f69d66@mail.gmail.com>
Jon Smirl wrote:
> On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
>> But aside from this point, I think an intrinsic part of implementing
>> incremental conversion is "convert the subsequent changes to the CVS
>> repository *subject to the constraints* imposed by decisions made in
>> earlier conversion runs. And the real trick is that things can be done
>> in CVS (e.g., line-end changes, manual copying of files in the repo)
>> that (a) are unversioned and (b) have retroactive effects that go
>> arbitrarily far back in time. This is the reason that I am pessimistic
>> that incremental conversion will ever work robustly.
>
> We don't need really robust incremental conversion. It just needs to
> work most of the time. Incremental conversion is usually used to track
> the main CVS repo with the new tool while people decide if they like
> the new tool. Commits will still flow to the CVS repo and get
> incrementally copied to the new tool so that it tracks CVS in close to
> real time.
I hadn't thought of the idea of using incremental conversion as an
advertising method for switching SCM systems :-) But if changes flow
back to CVS, doesn't this have to be pretty robust?
In our trial period, we simply did a single conversion to SVN and let
people play with this test repository. When we decided to switch over
we did another full conversion and simply discarded the changes that had
been made in the test SVN repository.
The use cases that I had considered were:
1. For conversions that take days, one could do a full commit while
leaving CVS online, then take CVS offline and do only an incremental
conversion to reduce SCM downtime. This is of course less of an issue
if you could bring the conversion time down to a couple hours for even
the largest CVS repos.
2. Long-term continuous mirroring (backwards and forwards) between CVS
and another SCM, to allow people to use their preferred tool. (I
actually think that this is a silly idea, but some people seem to like it.)
For both of these applications, incremental conversion would have to be
robust (for 1 it would at least have to give a clear indication of
unrecoverable errors).
Michael
^ permalink raw reply
* Re: cvs import
From: Martin Langhoff @ 2006-09-14 4:40 UTC (permalink / raw)
To: Michael Haggerty
Cc: Markus Schiltknecht, Jon Smirl, Git Mailing List, monotone-devel,
dev
In-Reply-To: <4508D7DA.8000302@alum.mit.edu>
On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> > IIRC, it places branch tags as late as possible. I haven't looked at
> > it in detail, but an import immediately after the first commit against
> > the branch may yield a different branchpoint from the same import done
> > a bit later.
>
> This is correct. And IMO it makes sense from the standpoint of an
> all-at-once conversion.
>
> But I was under the impression that this wouldn't matter for
> content-indexed-based SCMs. The content of all possible branching
> points is identical, and therefore from your point of view the topology
> should be the same, no?
Exactly. But if you shift the branching point to later, two things change
- it is possible that (in some corner cases) the content itself
changes as the branching point could end up being moved a couple of
commits "later". one of the downsides of cvs not being atomic.
- even if the content does not change, rearranging of history in git
is a no-no. git relies on history being read-only 100%
> But aside from this point, I think an intrinsic part of implementing
> incremental conversion is "convert the subsequent changes to the CVS
> repository *subject to the constraints* imposed by decisions made in
> earlier conversion runs.
Yes, and that's a fundamental change in the algorithm. That's exactly
why I mentioned it in this thread ;-) Any incremental importer has to
make up some parts of history, and then remember what it has made up.
So part of the process becomes
- figure our history on top of the history we already parsed
- check whether the cvs repo now has any 'new' history that affects
already-parsed history negatively, and report those as errors
hmmmmmm.
> This is the reason that I am pessimistic
> that incremental conversion will ever work robustly.
We all are :) But for a repo that doesn't go through direct tampering,
we can improve the algorithm to be more stable.
martin
^ permalink raw reply
* Re: cvs import
From: Jon Smirl @ 2006-09-14 4:34 UTC (permalink / raw)
To: Michael Haggerty
Cc: Martin Langhoff, Markus Schiltknecht, Git Mailing List,
monotone-devel, dev
In-Reply-To: <4508D7DA.8000302@alum.mit.edu>
On 9/14/06, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> But aside from this point, I think an intrinsic part of implementing
> incremental conversion is "convert the subsequent changes to the CVS
> repository *subject to the constraints* imposed by decisions made in
> earlier conversion runs. And the real trick is that things can be done
> in CVS (e.g., line-end changes, manual copying of files in the repo)
> that (a) are unversioned and (b) have retroactive effects that go
> arbitrarily far back in time. This is the reason that I am pessimistic
> that incremental conversion will ever work robustly.
We don't need really robust incremental conversion. It just needs to
work most of the time. Incremental conversion is usually used to track
the main CVS repo with the new tool while people decide if they like
the new tool. Commits will still flow to the CVS repo and get
incrementally copied to the new tool so that it tracks CVS in close to
real time.
If the increment import messes up you can always redo a full import,
but a full Mozilla import takes about 2 hours with the git tools. I
would always do a full import on the day of the actual cut over.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: cvs import
From: Michael Haggerty @ 2006-09-14 4:17 UTC (permalink / raw)
To: Martin Langhoff
Cc: Markus Schiltknecht, Jon Smirl, Git Mailing List, monotone-devel,
dev
In-Reply-To: <46a038f90609131416s1a53b53xd12c3661140fec7a@mail.gmail.com>
Martin Langhoff wrote:
> On 9/14/06, Markus Schiltknecht <markus@bluegap.ch> wrote:
>> Martin Langhoff wrote:
>> > On 9/14/06, Jon Smirl <jonsmirl@gmail.com> wrote:
>> >> Let's copy the git list too and maybe we can come up with one importer
>> >> for everyone.
>> >
>> > It's a really good idea. cvsps has been for a while a (limited, buggy)
>> > attempt at that. One thing that bothers me in the cvs2svn algorithm is
>> > that is not stable in its decisions about where the branching point is
>> > -- run the import twice at different times and it may tell you that
>> > the branching point has moved.
>>
>> Huh? Really? Why is that? I don't see reasons for such a thing happening
>> when studying the algorithm.
>>
>> For sure the proposed dependency-resolving algorithm which does not rely
>> on timestamps does not have that problem.
>
> IIRC, it places branch tags as late as possible. I haven't looked at
> it in detail, but an import immediately after the first commit against
> the branch may yield a different branchpoint from the same import done
> a bit later.
This is correct. And IMO it makes sense from the standpoint of an
all-at-once conversion.
But I was under the impression that this wouldn't matter for
content-indexed-based SCMs. The content of all possible branching
points is identical, and therefore from your point of view the topology
should be the same, no?
But aside from this point, I think an intrinsic part of implementing
incremental conversion is "convert the subsequent changes to the CVS
repository *subject to the constraints* imposed by decisions made in
earlier conversion runs. And the real trick is that things can be done
in CVS (e.g., line-end changes, manual copying of files in the repo)
that (a) are unversioned and (b) have retroactive effects that go
arbitrarily far back in time. This is the reason that I am pessimistic
that incremental conversion will ever work robustly.
Michael
^ permalink raw reply
* Re: Marking abandoned branches
From: Sam Vilain @ 2006-09-14 3:37 UTC (permalink / raw)
To: Jon Smirl; +Cc: Petr Baudis, Git Mailing List
In-Reply-To: <9e4733910609130859v347a7a9ew5c3ebc982bf9b07b@mail.gmail.com>
Jon Smirl wrote:
> It is a historical import. Everything that was in the initial repo
> needs to be preserved otherwise they aren't going to get rid of the
> old CVS repo.
>
You can tag a tree which contains the raw RCS files used for the import;
then they'll be available forever!
Sam
^ permalink raw reply
* Re: cvs import
From: Daniel Carosone @ 2006-09-14 3:19 UTC (permalink / raw)
To: Shawn Pearce
Cc: Daniel Carosone, Keith Packard, monotone-devel, Jon Smirl, dev,
Git Mailing List
In-Reply-To: <20060914023017.GA31889@spearce.org>
[-- Attachment #1.1: Type: text/plain, Size: 1905 bytes --]
On Wed, Sep 13, 2006 at 10:30:17PM -0400, Shawn Pearce wrote:
> I don't know exactly how big it is but the Gentoo CVS repository
> is also considered to be very large (about the size of the Mozilla
> repository) and just as difficult to import. Its either crashed or
> taken about a month to process with the current Git CVS->Git tools.
Ah, thanks for the tip.
> Since I know that the bulk of the Gentoo CVS repository is the
> portage tree I did a quick find|wc -l in my /usr/portage; its about
> 124,500 files.
>
> Its interesting that Gentoo has almost as large of a repository given
> that its such a young project, compared to NetBSD and Mozilla. :-)
Portage uses files and thus CVS very differently, though. Each ebuild
for each package revision of each version of a third-party package
(like, say, monotone 0.28 and 0.29, and -r1, -r2 pkg bumps of those if
they were needed) is its own file that's added, maybe edited a couple
of times, and then deleted again later as new versions are added and
older ones retired. These are copies and renames in the workspace,
but are invisible to CVS. This uses up lots more files than a single
long-lived build that gets edited each time; the Attic dirs must have
huge numbers of files, way beyond the number that are live now.
This lets portage keep builds around in a HEAD checkout for multiple
versions at once, tagged internally with different statuses.
Effectively, these tags take the place of VCS-based branches and
releases, and are more flexible for end users tracking their favourite
applications while keeping the rest of their system stable.
If they had a VCS that supported file cloning and/or renaming, and
used that to follow history between these ebuild files, things would
be very different. There are some interesting use cases for VCS tools
in supporting this behaviour nicely, too.
--
Dan.
[-- Attachment #1.2: Type: application/pgp-signature, Size: 186 bytes --]
[-- Attachment #2: Type: text/plain, Size: 158 bytes --]
_______________________________________________
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
^ permalink raw reply
* Re: [Monotone-devel] cvs import
From: Shawn Pearce @ 2006-09-14 2:35 UTC (permalink / raw)
To: Keith Packard
Cc: Nathaniel Smith, Markus Schiltknecht, monotone-devel, dev,
Git Mailing List
In-Reply-To: <1158190921.29313.175.camel@neko.keithp.com>
Keith Packard <keithp@keithp.com> wrote:
> On Wed, 2006-09-13 at 15:52 -0700, Nathaniel Smith wrote:
>
> > Regarding the basic dependency-based algorithm, the approach of
> > throwing everything into blobs and then trying to tease them apart
> > again seems backwards. What I'm thinking is, first we go through and
> > build the history graph for each file. Now, advance a frontier across
> > the all of these graphs simultaneously. Your frontier is basically a
> > map <filename -> CVS revision>, that represents a tree snapshot.
>
> Parsecvs does this, except backwards from now into the past; I found it
> easier to identify merge points than branch points (Oh, look, these two
> branches are the same now, they must have merged).
Why not let Git do that? If two branches are the same in CVS then
shouldn't they have the same tree SHA1 in Git? Surely comparing
20 bytes of SHA1 is faster than almost any other comparsion...
> However, this means that parsecvs must hold the entire tree state in
> memory, which turned out to be its downfall with large repositories.
> Worked great for all of X.org, not so good with Mozilla.
Any chance that can be paged in on demand from some sort of work
file? git-fast-import hangs onto a configurable number of tree
states (default of 5) but keeps them in an LRU chain and dumps the
ones that aren't current.
--
Shawn.
^ permalink raw reply
* Re: [Monotone-devel] cvs import
From: Shawn Pearce @ 2006-09-14 2:30 UTC (permalink / raw)
To: Daniel Carosone
Cc: Jon Smirl, Keith Packard, dev, monotone-devel, Git Mailing List
In-Reply-To: <20060914015324.GX29625@bcd.geek.com.au>
Daniel Carosone <dan@geek.com.au> wrote:
> On Wed, Sep 13, 2006 at 08:57:33PM -0400, Jon Smirl wrote:
> > Mozilla is 120,000 files. The complexity comes from 10 years worth of
> > history. A few of the files have around 1,700 revisions. There are
> > about 1,600 branches and 1,000 tags. The branch number is inflated
> > because cvs2svn is generating extra branches, the real number is
> > around 700. The CVS repo takes 4.2GB disk space. cvs2svn turns this
> > into 250,000 commits over about 1M unique revisions.
>
> Those numbers are pretty close to those in the NetBSD repository, and
> between them these probably represent just about the most extensive
> public CVS test data available.
I don't know exactly how big it is but the Gentoo CVS repository
is also considered to be very large (about the size of the Mozilla
repository) and just as difficult to import. Its either crashed or
taken about a month to process with the current Git CVS->Git tools.
Since I know that the bulk of the Gentoo CVS repository is the
portage tree I did a quick find|wc -l in my /usr/portage; its about
124,500 files.
Its interesting that Gentoo has almost as large of a repository given
that its such a young project, compared to NetBSD and Mozilla. :-)
--
Shawn.
^ permalink raw reply
* [PATCH] Trivial support for cloning and fetching via ftp://.
From: Sasha Khapyorsky @ 2006-09-14 2:24 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
This adds trivial support for cloning and fetching via ftp://.
Signed-off-by: Sasha Khapyorsky <sashak@voltaire.com>
---
git-clone.sh | 2 +-
git-fetch.sh | 4 ++--
git-ls-remote.sh | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/git-clone.sh b/git-clone.sh
index 7060bda..e1b3bf3 100755
--- a/git-clone.sh
+++ b/git-clone.sh
@@ -298,7 +298,7 @@ yes,yes)
fi
git-ls-remote "$repo" >"$GIT_DIR/CLONE_HEAD" || exit 1
;;
- https://*|http://*)
+ https://*|http://*|ftp://*)
if test -z "@@NO_CURL@@"
then
clone_dumb_http "$repo" "$D"
diff --git a/git-fetch.sh b/git-fetch.sh
index c2eebee..09a5d6c 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -286,7 +286,7 @@ fetch_main () {
# There are transports that can fetch only one head at a time...
case "$remote" in
- http://* | https://*)
+ http://* | https://* | ftp://*)
if [ -n "$GIT_SSL_NO_VERIFY" ]; then
curl_extra_args="-k"
fi
@@ -350,7 +350,7 @@ fetch_main () {
done
case "$remote" in
- http://* | https://* | rsync://* )
+ http://* | https://* | ftp://* | rsync://* )
;; # we are already done.
*)
( : subshell because we muck with IFS
diff --git a/git-ls-remote.sh b/git-ls-remote.sh
index 2fdcaf7..2c0b521 100755
--- a/git-ls-remote.sh
+++ b/git-ls-remote.sh
@@ -49,7 +49,7 @@ trap "rm -fr $tmp-*" 0 1 2 3 15
tmpdir=$tmp-d
case "$peek_repo" in
-http://* | https://* )
+http://* | https://* | ftp://* )
if [ -n "$GIT_SSL_NO_VERIFY" ]; then
curl_extra_args="-k"
fi
--
1.4.2.gffe87-dirty
^ permalink raw reply related
* Re: cvs import
From: Daniel Carosone @ 2006-09-14 1:53 UTC (permalink / raw)
To: Jon Smirl; +Cc: dev, Keith Packard, monotone-devel, Git Mailing List
In-Reply-To: <9e4733910609131757l7ce4b637oae18b523b1b7f0a4@mail.gmail.com>
[-- Attachment #1.1: Type: text/plain, Size: 1676 bytes --]
On Wed, Sep 13, 2006 at 08:57:33PM -0400, Jon Smirl wrote:
> Mozilla is 120,000 files. The complexity comes from 10 years worth of
> history. A few of the files have around 1,700 revisions. There are
> about 1,600 branches and 1,000 tags. The branch number is inflated
> because cvs2svn is generating extra branches, the real number is
> around 700. The CVS repo takes 4.2GB disk space. cvs2svn turns this
> into 250,000 commits over about 1M unique revisions.
Those numbers are pretty close to those in the NetBSD repository, and
between them these probably represent just about the most extensive
public CVS test data available.
I've only done imports of individual top-level dirs (what used to be
modules), like src and pkgsrc, because they're used independently and
don't really overlap.
src had about 180k commits over 1M versions of 120k files, 1000 tags
and 260 branches. pkgsrc had 110k commits over about half as many
files and versions thereof. We too have a few hot files, one had
13,625 revisions. xsrc adds a bunch more files and content, but not
many versions; that's mostly vendor branches and only some local
changes. Between them the cvs ,v files take up 4.7G covering about 13
years of history.
One thing that was interesting was that "src" used to be several
different modules, but we rearranged the repository at one point to
match the checkout structure these modules produced (combining them
all under the src dir). This doesn't seem to have upset the import at
all. Just about every other form of CVS evil has been perpetrated in
this repository at some stage or other too, but always very carefully.
--
Dan.
[-- Attachment #1.2: Type: application/pgp-signature, Size: 186 bytes --]
[-- Attachment #2: Type: text/plain, Size: 158 bytes --]
_______________________________________________
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
^ permalink raw reply
* Re: [PATCH 2/2] Handle invalid argc gently
From: Dmitry V. Levin @ 2006-09-14 1:31 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vu03b5k0t.fsf@assigned-by-dhcp.cox.net>
[-- Attachment #1: Type: text/plain, Size: 296 bytes --]
On Wed, Sep 13, 2006 at 06:12:02PM -0700, Junio C Hamano wrote:
> "Dmitry V. Levin" <ldv@altlinux.org> writes:
>
> > describe, git: Handle argc==0 case the same way as argc==1.
>
> When does one get (ac == 0)?
When one executes something like
execlp("git", NULL, NULL);
--
ldv
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [PATCH 2/2] Handle invalid argc gently
From: Junio C Hamano @ 2006-09-14 1:12 UTC (permalink / raw)
To: Dmitry V. Levin; +Cc: git
In-Reply-To: <20060914010409.GB20593@nomad.office.altlinux.org>
"Dmitry V. Levin" <ldv@altlinux.org> writes:
> describe, git: Handle argc==0 case the same way as argc==1.
When does one get (ac == 0)?
^ permalink raw reply
* [PATCH] Documentation: Fix broken links
From: Dmitry V. Levin @ 2006-09-14 1:04 UTC (permalink / raw)
To: Junio C Hamano; +Cc: GIT mailing list
core-tutorial.txt, cvs-migration.txt, tutorial-2.txt:
Fix broken links.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
---
Documentation/core-tutorial.txt | 2 +-
Documentation/cvs-migration.txt | 2 +-
Documentation/tutorial-2.txt | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/core-tutorial.txt b/Documentation/core-tutorial.txt
index 1185897..47505aa 100644
--- a/Documentation/core-tutorial.txt
+++ b/Documentation/core-tutorial.txt
@@ -1620,7 +1620,7 @@ suggested in the previous section may be
have to worry. git supports "shared public repository" style of
cooperation you are probably more familiar with as well.
-See link:cvs-migration.txt[git for CVS users] for the details.
+See link:cvs-migration.html[git for CVS users] for the details.
Bundling your work together
---------------------------
diff --git a/Documentation/cvs-migration.txt b/Documentation/cvs-migration.txt
index d2b0bd3..6812683 100644
--- a/Documentation/cvs-migration.txt
+++ b/Documentation/cvs-migration.txt
@@ -172,7 +172,7 @@ Advanced Shared Repository Management
Git allows you to specify scripts called "hooks" to be run at certain
points. You can use these, for example, to send all commits to the shared
-repository to a mailing list. See link:hooks.txt[Hooks used by git].
+repository to a mailing list. See link:hooks.html[Hooks used by git].
You can enforce finer grained permissions using update hooks. See
link:howto/update-hook-example.txt[Controlling access to branches using
diff --git a/Documentation/tutorial-2.txt b/Documentation/tutorial-2.txt
index 2f4fe12..42b6e7d 100644
--- a/Documentation/tutorial-2.txt
+++ b/Documentation/tutorial-2.txt
@@ -368,7 +368,7 @@ in the index file is identical to the on
In addition to being the staging area for new commits, the index file
is also populated from the object database when checking out a
branch, and is used to hold the trees involved in a merge operation.
-See the link:core-tutorial.txt[core tutorial] and the relevant man
+See the link:core-tutorial.html[core tutorial] and the relevant man
pages for details.
What next?
--
ldv
^ permalink raw reply related
* [PATCH 1/2] Make more commands work in subdirectory
From: Dmitry V. Levin @ 2006-09-14 1:03 UTC (permalink / raw)
To: Junio C Hamano; +Cc: GIT mailing list
count-objects, describe, merge-tree:
Call setup_git_directory() to make these commands work in subdirectory.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
---
describe.c | 2 ++
git.c | 2 +-
merge-tree.c | 2 ++
3 files changed, 5 insertions(+), 1 deletions(-)
diff --git a/describe.c b/describe.c
index 5dd8b2e..5ed052d 100644
--- a/describe.c
+++ b/describe.c
@@ -161,6 +161,8 @@ int main(int argc, char **argv)
usage(describe_usage);
}
+ setup_git_directory();
+
if (i == argc)
describe("HEAD", 1);
else
diff --git a/git.c b/git.c
index 335f405..47c85e1 100644
--- a/git.c
+++ b/git.c
@@ -224,7 +224,7 @@ static void handle_internal_command(int
{ "checkout-index", cmd_checkout_index, RUN_SETUP },
{ "check-ref-format", cmd_check_ref_format },
{ "commit-tree", cmd_commit_tree, RUN_SETUP },
- { "count-objects", cmd_count_objects },
+ { "count-objects", cmd_count_objects, RUN_SETUP },
{ "diff", cmd_diff, RUN_SETUP },
{ "diff-files", cmd_diff_files, RUN_SETUP },
{ "diff-index", cmd_diff_index, RUN_SETUP },
diff --git a/merge-tree.c b/merge-tree.c
index 60df758..c154dcf 100644
--- a/merge-tree.c
+++ b/merge-tree.c
@@ -340,6 +340,8 @@ int main(int argc, char **argv)
if (argc < 4)
usage(merge_tree_usage);
+ setup_git_directory();
+
buf1 = get_tree_descriptor(t+0, argv[1]);
buf2 = get_tree_descriptor(t+1, argv[2]);
buf3 = get_tree_descriptor(t+2, argv[3]);
--
ldv
^ permalink raw reply related
* [PATCH 2/2] Handle invalid argc gently
From: Dmitry V. Levin @ 2006-09-14 1:04 UTC (permalink / raw)
To: Junio C Hamano; +Cc: GIT mailing list
describe, git: Handle argc==0 case the same way as argc==1.
merge-tree: Refuse excessive arguments.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
---
describe.c | 2 +-
git.c | 2 +-
merge-tree.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/describe.c b/describe.c
index 5ed052d..2172e8f 100644
--- a/describe.c
+++ b/describe.c
@@ -163,7 +163,7 @@ int main(int argc, char **argv)
setup_git_directory();
- if (i == argc)
+ if (i >= argc)
describe("HEAD", 1);
else
while (i < argc) {
diff --git a/git.c b/git.c
index 47c85e1..8c182a5 100644
--- a/git.c
+++ b/git.c
@@ -294,7 +294,7 @@ static void handle_internal_command(int
int main(int argc, const char **argv, char **envp)
{
- const char *cmd = argv[0];
+ const char *cmd = argv[0] ? argv[0] : "git-help";
char *slash = strrchr(cmd, '/');
const char *exec_path = NULL;
int done_alias = 0;
diff --git a/merge-tree.c b/merge-tree.c
index c154dcf..692ede0 100644
--- a/merge-tree.c
+++ b/merge-tree.c
@@ -337,7 +337,7 @@ int main(int argc, char **argv)
struct tree_desc t[3];
void *buf1, *buf2, *buf3;
- if (argc < 4)
+ if (argc != 4)
usage(merge_tree_usage);
setup_git_directory();
--
ldv
^ permalink raw reply related
* Re: [Monotone-devel] cvs import
From: Jon Smirl @ 2006-09-14 0:57 UTC (permalink / raw)
To: Keith Packard, dev, monotone-devel, Git Mailing List
In-Reply-To: <20060914003242.GA19228@frances.vorpus.org>
On 9/13/06, Nathaniel Smith <njs@pobox.com> wrote:
> On Wed, Sep 13, 2006 at 04:42:01PM -0700, Keith Packard wrote:
> > However, this means that parsecvs must hold the entire tree state in
> > memory, which turned out to be its downfall with large repositories.
> > Worked great for all of X.org, not so good with Mozilla.
>
> Does anyone know how big Mozilla (or other humonguous repos, like KDE)
> are, in terms of number of files?
Mozilla is 120,000 files. The complexity comes from 10 years worth of
history. A few of the files have around 1,700 revisions. There are
about 1,600 branches and 1,000 tags. The branch number is inflated
because cvs2svn is generating extra branches, the real number is
around 700. The CVS repo takes 4.2GB disk space. cvs2svn turns this
into 250,000 commits over about 1M unique revisions.
>
> A few numbers for repositories I had lying around:
> Linux kernel -- ~21,000
> gcc -- ~42,000
> NetBSD "src" repo -- ~100,000
> uClinux distro -- ~110,000
>
> These don't seem very indimidating... even if it takes an entire
> kilobyte per CVS revision to store the information about it that we
> need to make decisions about how to move the frontier... that's only
> 110 megabytes for the largest of these repos. The frontier sweeping
> algorithm only _needs_ to have available the current frontier, and the
> current frontier+1. Storing information on every version of every
> file in memory might be worse; but since the algorithm accesses this
> data in a linear way, it'd be easy enough to stick those in a
> lookaside table on disk if really necessary, like a bdb or sqlite file
> or something.
>
> (Again, in practice storing all the metadata for the entire 180k
> revisions of the 100k files in the netbsd repo was possible on a
> desktop. Monotone's cvs_import does try somewhat to be frugal about
> memory, though, interning strings and suchlike.)
>
> -- Nathaniel
>
> --
> When the flush of a new-born sun fell first on Eden's green and gold,
> Our father Adam sat under the Tree and scratched with a stick in the mould;
> And the first rude sketch that the world had seen was joy to his mighty heart,
> Till the Devil whispered behind the leaves, "It's pretty, but is it Art?"
> -- The Conundrum of the Workshops, Rudyard Kipling
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: cvs import
From: Nathaniel Smith @ 2006-09-14 0:32 UTC (permalink / raw)
To: Keith Packard; +Cc: dev, monotone-devel, Git Mailing List
In-Reply-To: <1158190921.29313.175.camel@neko.keithp.com>
On Wed, Sep 13, 2006 at 04:42:01PM -0700, Keith Packard wrote:
> However, this means that parsecvs must hold the entire tree state in
> memory, which turned out to be its downfall with large repositories.
> Worked great for all of X.org, not so good with Mozilla.
Does anyone know how big Mozilla (or other humonguous repos, like KDE)
are, in terms of number of files?
A few numbers for repositories I had lying around:
Linux kernel -- ~21,000
gcc -- ~42,000
NetBSD "src" repo -- ~100,000
uClinux distro -- ~110,000
These don't seem very indimidating... even if it takes an entire
kilobyte per CVS revision to store the information about it that we
need to make decisions about how to move the frontier... that's only
110 megabytes for the largest of these repos. The frontier sweeping
algorithm only _needs_ to have available the current frontier, and the
current frontier+1. Storing information on every version of every
file in memory might be worse; but since the algorithm accesses this
data in a linear way, it'd be easy enough to stick those in a
lookaside table on disk if really necessary, like a bdb or sqlite file
or something.
(Again, in practice storing all the metadata for the entire 180k
revisions of the 100k files in the netbsd repo was possible on a
desktop. Monotone's cvs_import does try somewhat to be frugal about
memory, though, interning strings and suchlike.)
-- Nathaniel
--
When the flush of a new-born sun fell first on Eden's green and gold,
Our father Adam sat under the Tree and scratched with a stick in the mould;
And the first rude sketch that the world had seen was joy to his mighty heart,
Till the Devil whispered behind the leaves, "It's pretty, but is it Art?"
-- The Conundrum of the Workshops, Rudyard Kipling
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox