* [PATCH] Add a "--notags" option for git-p4import.
From: Sean @ 2006-06-15 21:26 UTC (permalink / raw)
To: git
P4import currently creates a git tag for every commit it imports.
When importing from a large repository too many tags can be created
for git to manage, so this provides an option to shut that feature
off if necessary.
Signed-off-by: Sean Estabrooks <seanlkml@sympatico.ca>
---
Documentation/git-p4import.txt | 5 ++++-
git-p4import.py | 12 ++++++++----
2 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/Documentation/git-p4import.txt b/Documentation/git-p4import.txt
index c198ff2..0858e5e 100644
--- a/Documentation/git-p4import.txt
+++ b/Documentation/git-p4import.txt
@@ -8,7 +8,7 @@ git-p4import - Import a Perforce reposit
SYNOPSIS
--------
-`git-p4import` [-q|-v] [--authors <file>] [-t <timezone>] <//p4repo/path> <branch>
+`git-p4import` [-q|-v] [--notags] [--authors <file>] [-t <timezone>] <//p4repo/path> <branch>
`git-p4import` --stitch <//p4repo/path>
@@ -43,6 +43,9 @@ OPTIONS
Specify an authors file containing a mapping of Perforce user
ids to full names and email addresses (see Notes below).
+\--notags::
+ Do not create a tag for each imported commit.
+
\--stitch::
Import the contents of the given perforce branch into the
currently checked out git branch.
diff --git a/git-p4import.py b/git-p4import.py
index 74172ab..908941d 100644
--- a/git-p4import.py
+++ b/git-p4import.py
@@ -23,7 +23,6 @@ s = signal(SIGINT, SIG_DFL)
if s != default_int_handler:
signal(SIGINT, s)
-
def die(msg, *args):
for a in args:
msg = "%s %s" % (msg, a)
@@ -38,6 +37,7 @@ verbosity = 1
logfile = "/dev/null"
ignore_warnings = False
stitch = 0
+tagall = True
def report(level, msg, *args):
global verbosity
@@ -261,10 +261,9 @@ class git_command:
self.make_tag("p4/%s"%id, commit)
self.git("update-ref HEAD %s %s" % (commit, current) )
-
try:
opts, args = getopt.getopt(sys.argv[1:], "qhvt:",
- ["authors=","help","stitch=","timezone=","log=","ignore"])
+ ["authors=","help","stitch=","timezone=","log=","ignore","notags"])
except getopt.GetoptError:
usage()
@@ -275,6 +274,8 @@ for o, a in opts:
verbosity += 1
if o in ("--log"):
logfile = a
+ if o in ("--notags"):
+ tagall = False
if o in ("-h", "--help"):
usage()
if o in ("--ignore"):
@@ -350,7 +351,10 @@ for id in changes:
report(1, "Importing changeset", id)
change = p4.describe(id)
p4.sync(id)
- git.commit(change.author, change.email, change.date, change.msg, id)
+ if tagall :
+ git.commit(change.author, change.email, change.date, change.msg, id)
+ else:
+ git.commit(change.author, change.email, change.date, change.msg, "import")
if stitch == 1:
git.clean_directories()
stitch = 0
--
1.4.0.rc2
^ permalink raw reply related
* Re: observations on parsecvs testing
From: Keith Packard @ 2006-06-15 22:03 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: keithp, git
In-Reply-To: <Pine.LNX.4.64.0606151529350.16002@localhost.localdomain>
[-- Attachment #1: Type: text/plain, Size: 3871 bytes --]
On Thu, 2006-06-15 at 16:37 -0400, Nicolas Pitre wrote:
> My machine is a P4 @ 3GHz with 1GB ram.
>
> Feeding parsecvs with the Mozilla repository, it first ran for 175
> minutes with about 98% CPU spent in user space reading the 100458 ,v
> files and writing 700000+ blob objects. Memory usage grew to 1789MB
> total while the resident memory saturated around 700MB. This part was
> fine even with 1GB of ram since unused memory was gently pushed to swap.
> Only problem is that spawned git-pack-object instances started failing
> with memory allocation by that time, which is unffortunate but not
> fatal.
Right, the ,v -> blob conversion process uses around 160 bytes per
revision as best I can count (one rev_commit, one rev_file and
a 41-byte sha1 string); 700000 revisions would therefore use 1.1GB just
for the revision objects. It should be possible to reduce the size of
this data structure fairly significantly; converting the sha1 value to
binary and compressing the CVS revision number to minimal length.
Switching from the general git/cvs structure to this cvs-specific
structure is 'on the list' of things I'd like to do.
> But then things started to go bad after all ,v files were parsed. The
> parsecvs dropped to 3% CPU while the rest of the time was spent waiting
> after swap IO and therefore no substantial progress was made at that
> point.
Yeah, after this point, parsecvs is merging the computed revision
historys of the individual files into a global history. This means it's
walking across the whole set of files to compute each git commit. For
each branch, it computes the set of files visible at the head of that
branch and then sorts the last revision of the visible files to discover
the last change set along that branch, constructing a commit for each
logical changeset backwards from the present into the past. As it's
constructing commits from the present backwards, it must go all the way
to the past before it can emit any commits to the repository. So, it has
to save them somewhere; right now, it's saving them in memory. What it
could do is construct tree objects for each commit, saving only the sha1
that results and dump the rest of the data. That should save plenty of
memory, but would require a radical restructuring of the code (which is
desparately needed, btw). With this change, parsecvs should actually
*shrink* over time, instead of grow.
> So the Mozilla clearly requires 2GB of ram to realistically be converted
> to GIT using parsecvs, unless its second phase is reworked to avoid
> totally random access in memory in order to improve swap behavior, or
> its in-memory data set is shrinked at least by half.
Changing the data structures used in the first phase will shrink them
significantly; replacing the second state data structures with sha1 tree
hash values and disposing of the first phase objects incrementally
should elicit a shrinking memory pattern rather than growing. It might
well be easier at this point to just take the basic CVS parser and start
afresh though; the code is a horror show of incremental refinements.
> Also rcs2git() is very inefficient especially with files having many
> revisions as it reconstructs the delta chain on every call. For example
> mozilla/configure,v has at least 1690 revisions, and actually converting
> it into GIT blobs goes at a rate of 2.4 objects per second _only_ on my
> machine. Can't objects be created as the delta list is walked/applied
> instead? That would significantly reduce the initial convertion time.
Yes, I wanted to do this, but also wanted to ensure that the constructed
versions exactly matched the native rcs output. Starting with 'real' rcs
code seemed likely to ensure the latter. This "should" be easy to fix...
--
keith.packard@intel.com
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: observations on parsecvs testing
From: Keith Packard @ 2006-06-15 22:04 UTC (permalink / raw)
To: Sean; +Cc: keithp, Nicolas Pitre, git
In-Reply-To: <20060615164742.570e33a0.seanlkml@sympatico.ca>
[-- Attachment #1: Type: text/plain, Size: 477 bytes --]
On Thu, 2006-06-15 at 16:47 -0400, Sean wrote:
> las,
>
> That was a planned optimization which I did mention to Keith previously.
> Was kinda waiting to hear back how it was working for him, and if there
> was an interest to put more work into it to include in his mainline.
The rcs2git code is working great and is on 'master' at this point;
optimizations to generate all of the revisions in one pass would be
greatly appreciated.
--
keith.packard@intel.com
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: Autoconf/Automake
From: Yann Dirson @ 2006-06-15 22:05 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Phil Richards, git
In-Reply-To: <Pine.LNX.4.63.0606152239270.7480@wbgn013.biozentrum.uni-wuerzburg.de>
On Thu, Jun 15, 2006 at 10:42:40PM +0200, Johannes Schindelin wrote:
> As for now, I fail to see why the current system is not adequate for git!
I can reassure you, gazillions of people still fail to see why cvs is
not adequate for their project. And the ratio of devs in the
corporate world not knowning git to those not knowning cvs is far
superior to 2. And everyone here knows cvs is not more adequate than
git for so many tasks :)
Best regards,
--
Yann Dirson <ydirson@altern.org> |
Debian-related: <dirson@debian.org> | Support Debian GNU/Linux:
| Freedom, Power, Stability, Gratis
http://ydirson.free.fr/ | Check <http://www.debian.org/>
^ permalink raw reply
* Re: Autoconf/Automake
From: Linus Torvalds @ 2006-06-15 22:54 UTC (permalink / raw)
To: Yann Dirson; +Cc: Alex Riesen, Pavel Roskin, git
In-Reply-To: <20060615211454.GK7766@nowhere.earth>
On Thu, 15 Jun 2006, Yann Dirson wrote:
>
> In tha case of jam, the doc issue can certainly be raised, but the
> most prominent problem is probably that everyone and their dog knows
> make,
Oh, I agree. A "simpler" thing that people don't know is often much
inferior to a complex thing that people are generally intimately familiar
with.
I just personally believe that autoconf/automake are the worst of both
worlds (ie it's a _complex_ thing that a lot of people don't know).
GNU make in many ways is actually not that bad. Yeah, the makefiles get
more complex, but it's usually not totally unreadable, and you can do some
clever stuff with it.
The kernel makefiles are a pretty extreme example (and it hides a lot of
the complexity in files that get included and that most people never ever
need to look at). I suspect that git could more easily do something like
that (on a _much_ smaller scale - don't get me wrong).
Linus
^ permalink raw reply
* Re: Autoconf/Automake
From: Johannes Schindelin @ 2006-06-15 22:58 UTC (permalink / raw)
To: Yann Dirson; +Cc: Phil Richards, git
In-Reply-To: <20060615220534.GL7766@nowhere.earth>
Hi,
On Fri, 16 Jun 2006, Yann Dirson wrote:
> On Thu, Jun 15, 2006 at 10:42:40PM +0200, Johannes Schindelin wrote:
> > As for now, I fail to see why the current system is not adequate for git!
>
> I can reassure you, gazillions of people still fail to see why cvs is
> not adequate for their project. And the ratio of devs in the
> corporate world not knowning git to those not knowning cvs is far
> superior to 2. And everyone here knows cvs is not more adequate than
> git for so many tasks :)
You know as well as I that this comparison is unfair. I am _NOT_ a
corporate person. I hope that you do not judge me as a complete airhead.
The point is: the right tool solves the problem. You can have a tool which
is mighty cool, but way too powerful (AKA complicated).
As for CVS: there _are_ a few use cases where CVS is just the right tool.
There are many more use cases where git is more than adequate, where CVS
is not.
_BUT_: there are cases where something like autoconf/jam/cmake/blablabla
is adequate, but I still fail to see why for git, the makefile system
should not work. It is the most transparent way to configure a make system
I encountered. It is short, concise, and does the job. And I understand
it. As opposed to autoconf/jam/cmake/blablabla.
Hth,
Dscho
^ permalink raw reply
* Re: Autoconf/Automake
From: Johannes Schindelin @ 2006-06-15 23:10 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Yann Dirson, Alex Riesen, Pavel Roskin, git
In-Reply-To: <Pine.LNX.4.64.0606151545050.5498@g5.osdl.org>
Hi,
On Thu, 15 Jun 2006, Linus Torvalds wrote:
> On Thu, 15 Jun 2006, Yann Dirson wrote:
> >
> > In tha case of jam, the doc issue can certainly be raised, but the
> > most prominent problem is probably that everyone and their dog knows
> > make,
>
> Oh, I agree. A "simpler" thing that people don't know is often much
> inferior to a complex thing that people are generally intimately familiar
> with.
>
> I just personally believe that autoconf/automake are the worst of both
> worlds (ie it's a _complex_ thing that a lot of people don't know).
>
> GNU make in many ways is actually not that bad. Yeah, the makefiles get
> more complex, but it's usually not totally unreadable, and you can do some
> clever stuff with it.
I can add to that with first-hand experience of ant and maven.
A whole sh*t-load of people think make is broken. It does not live up to
what they want, and it is slow.
And then they invent a _DISEASE_ like ant, which _does not begin_ to sport
the features of make.
In a project I am stuck in, maven is used. It tries -- of all things -- to
fix a few shortcomings of ant -- which was supposed to fix shortcomings of
make! And let's face it. Maven is complicated, slow as a dog lacking all
four feet, and it still does not do the things I can do in three lines
with make. It's a complete desaster.
So to keep the discussion on topic: tell me what you want to fix wrt the
current setup of git, and I'll try to fix it in less than 10 lines of make
code. If that is impossible, let's continue then and there with the
discussion about a switch to a newer, less tested, replacement of make,
okay?
Ciao,
Dscho
^ permalink raw reply
* Setting up git server?
From: lamikr @ 2006-06-15 23:19 UTC (permalink / raw)
To: git
Hi
I have git-repo cloned from the linux-omap-2.6 that we have used as a
base for our h6300 development.
Earlier we have kept our kernel in svn (sync between git-branches and
svn has happened about once in a month by using
traditional diff files...)
I have now pulled the server to "/repos/git/linux-omap-h6300-2.6" and
setup the /etc/xinetd.d/git-daemon by using docs in
http://www.kernel.org/pub/software/scm/git/docs/everyday.html
How can I now create the git url for this? For example something like
this: git://aragorn.kortex.jyu.fi/repos/git/linux-omap-h6300-2.6.git
Mika
^ permalink raw reply
* Re: Security problem
From: Junio C Hamano @ 2006-06-16 0:12 UTC (permalink / raw)
To: Alexander Litvinov; +Cc: git
In-Reply-To: <200606151709.22752.lan@academsoft.ru>
Alexander Litvinov <lan@academsoft.ru> writes:
> Why does not git-checkout check if file content match name of the object ?
Good point. We could do a few things:
- entry.c:write_entry() could validate after read_sha1_file().
- read_sha1_file() could do the checking; this has performance
implications, though.
Cloning over git aware protocols validate the objects coming
over the wire, so it may make sense to cheat and do the former,
so that we do not have to pay the validation cost every time we
access any object.
^ permalink raw reply
* Re: 2.6.17-rc6-mm2
From: Goo GGooo @ 2006-06-16 1:14 UTC (permalink / raw)
To: linux-kernel, git
In-Reply-To: <ef5305790606142040r5912ce58kf9f889c3d61b2cc0@mail.gmail.com>
On 6/15/06, Goo GGooo <googgooo@gmail.com> wrote:
> Andrew Morton wrote:
>
> > - To fetch an -mm tree using git, use (for example)
> >
> > git fetch git://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git
> > v2.6.16-rc2-mm1
>
> I'm not able to get -mm tree from GIT. In
> http://git.kernel.org/.../smurf/linux-trees.git/refs/tags/ I can see
> the most recent tags like v2.6.17-rc6-mm2 but cg-clone
> http://git.kernel.org/.../smurf/linux-trees.git gives me only
> 2.6.16-rc3 :(
>
> I tried "cg-fetch v2.6.17-rc6-mm2" which seemed to fetch some more
> tags, then played with git-checkout & friends but still can't get the
> most recent source tree.
All right, finally this worked out:
git pull rsync://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git \
tag v2.6.17-rc6-mm2
Strange enough with http:// instead of rsync:// I got some message
about nonexistent tag.
Now when I try git pull with http:// again it says the tree is up to
date. However with git:// it started downloading more things and tags.
That's confusing - I believed all protocols should behave the same way...?
Goo
^ permalink raw reply
* Re: Security problem
From: Linus Torvalds @ 2006-06-16 2:28 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Alexander Litvinov, git
In-Reply-To: <7vbqsuc60q.fsf@assigned-by-dhcp.cox.net>
On Thu, 15 Jun 2006, Junio C Hamano wrote:
>
> Alexander Litvinov <lan@academsoft.ru> writes:
>
> > Why does not git-checkout check if file content match name of the object ?
>
> Good point. We could do a few things:
I missed the original mail. What's the problem?
If this is about the remote end lying about the SHA1 name, it's a total
non-issue for any of the native protocols, since the native protocols
don't actually send SHA1 names at all, they just send the data (and we
re-create the SHA1 name on receipt).
So there's no way to have the name of an object not match its content,
unless you have actual corruption (which is for git-fsck-object to find,
not somethign that should slow down any normal operation), or if you use
one of the dumb protocols.
And if you use the dumb protocols, the data should probably be validated
_there_ (by fetch(), rather than anywhere else). And for "rsync", you
really don't have much choice apart from doing a full fsck, I suspect.
So I don't see the security issue, unless you don't trust the local
filesystem, in which case nothing git can do matters at all..
Linus
^ permalink raw reply
* Re: 2.6.17-rc6-mm2
From: Linus Torvalds @ 2006-06-16 2:46 UTC (permalink / raw)
To: Goo GGooo; +Cc: linux-kernel, git
In-Reply-To: <ef5305790606151814i252c37c4mdd005f11f06ceac@mail.gmail.com>
On Fri, 16 Jun 2006, Goo GGooo wrote:
>
> That's confusing - I believed all protocols should behave the same way...?
Not really. The primary protocol is the native git one, and the others try
to do a best effort, but the http protocol really can't do a very good
job unless the server side has run "git update-server-info" to help the
http client along.
I suspect that the -mm git tree simply doesn't do that. In fact, even the
main tree didn't use to do it, but I finally just broke down and added the
proper hook to make it always do it automatically when I push.
(In case Andrew wants to do that, the way to do it is:
echo -e "#!/bin/sh\nexec git-update-server-info" > hooks/post-update
chmod +x hooks/post-update
inside the git repository - all it will do is always execute that script,
and this "git-update-server-info", after you've updated the repo).
Finally, the rsync protocol just copies all objects over, and since it
doesn't even know _which_ objects it is getting, it doesn't do the normal
tag following that the native git protocol does.
So to recap:
- http is fundamentally weaker, and needs some server-side help to work
- rsync is fine for the initial clone, but doesn't actually know what
it's doing, so the end result can actually even be a corrupted
repository, because you happened to rsync just as it was updating.
- the native git protocol generally should be considered the golden
standard, where the other ones are just fallbacks in case of problems
(like firewalls that don't let git:// through, or more commonly hosted
servers that don't do the git protocol at all).
Which hopefully clarifies the issue a bit.
Linus
^ permalink raw reply
* Re: Security problem
From: Linus Torvalds @ 2006-06-16 2:56 UTC (permalink / raw)
To: Alexander Litvinov; +Cc: Junio C Hamano, git
In-Reply-To: <200606160931.29553.lan@academsoft.ru>
On Fri, 16 Jun 2006, Alexander Litvinov wrote:
>
> I have found the ability to hack git repo. After this hacking people will
> checkout hacked files from the "trusted" commit. Only git-fsck-objects will
> complain at this.
Right.
If you can't trust your local filesystem, you are screwed.
git-fsck-objects will notice when somebody has done something bad, but
> Why does not git-checkout check if file content match name of the object ?
Why would it? It really just slows things down, and if you don't trust
your local repo, people can "hack" you much more easily by just generating
a _proper_ tree with the _proper_ data, and git checkout checking the SHA1
wouldn't help at all.
The way to security lies in using git-fsck-objects, together with an
_external_ source of trust. For example, that external source of trust may
be a signed tag, or, perhaps even more simply, just by saving off the top
commit name on some trusted medium.
But you do need a "point of trust" to start with. Without that, it's a lot
easier to "hack" a git repo by doing
echo 'Hacked file' > a
git commit --amend a
git prune
and now the file "a" has changed to "Hacked file", and even
git-fsck-objects can't tell that anything bad happened.
(Btw, if you want to _hide_ the fact that "a" now contains "Hacked file",
you do so by faking it in the index. You can have the checked-out copy say
what it should say - ie "Usual file" - and if you don't want git to show
you the difference to HEAD, you edit the .git/index file by hand so that
the timestamp, size and inode matches the real SHA1, even though the
_contents_ match "Usual file").
See?
You do need to trust something. Normally you'd trust your own filesystem,
but git certainly supports other forms of trust through either the native
support for signed certificates in the form of tags, or any other form of
external trust.
Linus
^ permalink raw reply
* Re: Security problem
From: Alexander Litvinov @ 2006-06-16 3:54 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0606151948230.5498@g5.osdl.org>
> If you can't trust your local filesystem, you are screwed.
You are right, I trust my file system. But if our team had central repo with
ssh access to that machine, every developer can hack central repo.
Whould git-clone/git-fetch warn me about this ?
My own test with (another) local repo says:
lan@lan:~/tmp/git/test> git clone 1 2
Generating pack...
Done counting 3 objects.
Deltifying 3 objects.
100% (3/3) done
Total 3, written 3 (delta 0), reused 0 (delta 0)
error: git-checkout-index: unable to read sha1 file of a
(3609f20ebd357679b111783e8afaf36ec46427f3)
It can't checkout object (3609f20ebd357679b111783e8afaf36ec46427f3 is the
original file). It seems packed repos are safe from this point.
^ permalink raw reply
* Re: Security problem
From: Linus Torvalds @ 2006-06-16 5:00 UTC (permalink / raw)
To: Alexander Litvinov; +Cc: Junio C Hamano, git
In-Reply-To: <200606161054.46813.lan@academsoft.ru>
On Fri, 16 Jun 2006, Alexander Litvinov wrote:
>
> You are right, I trust my file system. But if our team had central repo with
> ssh access to that machine, every developer can hack central repo.
>
> Whould git-clone/git-fetch warn me about this ?
Using the native protocol, yes. Using rsync, unless you explicitly fsck
the result, no.
> It can't checkout object (3609f20ebd357679b111783e8afaf36ec46427f3 is the
> original file). It seems packed repos are safe from this point.
Well, they may not be "safe" - you just need to work a _lot_ harder to
corrupt a pack-file in any interesting manner. And again, git-fsck-objects
would pick up any such thing going on.
Anyway, what it boils down to is that anybody who has write access to a
particular repository can certainly change the repo in "interesting" ways.
However, there are various inherent safety valves in place that make it
really hard to corrupt on a bigger scale.
The first is that git-fsck-objects will definitely find any repository
inconsistency, and to get around that, you either have to get around the
basic properties of SHA-1 (ie break the hash) _or_ you have to actually
change the repository so that it's still a valid repo, just with different
content.
So let's take a look at those two cases:
- if you corrupt the repository, subsequent clones (or even pulls) from
the corrupt repository simply won't work if you use the native
protocol, because the native protocol doesn't actually trust anything
but the actual contents (so if the contents won't match, then neither
will the SHA1 names). So the corruption is pretty strictly limited to
the _one_ repository that the attacker had write access to.
So there's a pretty fundamental "corruption containment" part there.
(Side note: there's no question that we might well be able to do
better. A _malicious_ server could actually send a corrupt pack, and
it's possible that a properly corrupted remote archive could cause even
a "good" git-send-pack to just silently send a corrupt pack, so that
you'd need to use "git-fsck-objects" on the receiving side to notice
that you are missing objects, for example)
- if the repository is good (ie fsck is fine), then obviously a "git
pull" will also succeed. However, you can't _hide_ the data the way you
tried to do: when the receiver checks out the most recent version, it
will definitely use the data in the object, there's no way to get the
server to serve different data in objects and in the working tree
(because the server literally doesn't even send the working tree at
all).
So you can always convince somebody to pull from an "evil repository",
and that's no different from committing a bug by mistake. But at least
you can't try to hide the bug just in the object store and have it not
show up in diffs and in checked-out copies.
The latter case is true even with http and rsync, the actual pull event
always pulls just the database, never any checked-out state (in fact,
the common case is obviously to pull from a bare repository that doesn't
even _have_ checked-out state). So you can't hide things in the index or
in the checked-out state except in the filesystem that you have direct
write access to.
But yeah, I actually still personally do a fair number of
"git-fsck-objects". I've never found anything that way since very early on
(and back then, the real problem was rsync getting objects that weren't
reachable), but I still do it. It makes me feel happier.
Of course, bugs always happen. But I can pretty much guarantee that git is
fundamentally harder to corrupt than most things. We've had git-fsck-cache
since April 8th last year (or, put another way, literally since "Day 2" in
git terms - it's the eight commit in the whole git history).
Git also has an almost total lack of redundant information. There's
basically no "duplicate" information in the repository format itself where
you could hide something so that it wouldn't be noticed.
In a checked-out project, the checked-out state itself is "duplicate
information" (and that was where your "attack" tried to hide things), and
there's the index (which is actually a much better and subtle place to
hide things ;). But neither of them have any life outside of that
particular repository.
Linus
^ permalink raw reply
* Re: Security problem
From: Alexander Litvinov @ 2006-06-16 5:37 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0606152137410.5498@g5.osdl.org>
> Well, they may not be "safe" - you just need to work a _lot_ harder to
> corrupt a pack-file in any interesting manner. And again, git-fsck-objects
> would pick up any such thing going on.
As it shown in pack-objects.c, each object have stored sha1, almost the same
as file rename.
> The first is that git-fsck-objects will definitely find any repository
> inconsistency, and to get around that, you either have to get around the
> basic properties of SHA-1 (ie break the hash) _or_ you have to actually
> change the repository so that it's still a valid repo, just with different
> content.
I still belive SHA-1 is good enouth to hash files - I did not hear about
generation reasonable duplicate that can compile and work :-)
> - if you corrupt the repository, subsequent clones (or even pulls) from
> the corrupt repository simply won't work if you use the native
> protocol, because the native protocol doesn't actually trust anything
> but the actual contents (so if the contents won't match, then neither
> will the SHA1 names). So the corruption is pretty strictly limited to
> the _one_ repository that the attacker had write access to.
As I understand sent pack file will contains actial SHA-1 of objects. And any
hack will be cleary visible.
> So there's a pretty fundamental "corruption containment" part there.
...
Situation with evil repo is clear to me: you can turst only to trusted commit
identified by SHA-1
> But yeah, I actually still personally do a fair number of
> "git-fsck-objects". I've never found anything that way since very early on
> (and back then, the real problem was rsync getting objects that weren't
> reachable), but I still do it. It makes me feel happier.
As the result: Always fsck repo after pull/clone !
^ permalink raw reply
* Re: 2.6.17-rc6-mm2
From: Goo GGooo @ 2006-06-16 5:49 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux-kernel, git
In-Reply-To: <Pine.LNX.4.64.0606151937360.5498@g5.osdl.org>
On 6/16/06, Linus Torvalds <torvalds@osdl.org> wrote:
> So to recap:
> - http is fundamentally weaker, and needs some server-side help to work
> - rsync is fine for the initial clone, but doesn't actually know what
> it's doing, so the end result can actually even be a corrupted
> repository, because you happened to rsync just as it was updating.
> - the native git protocol generally should be considered the golden
> standard, where the other ones are just fallbacks in case of problems
> (like firewalls that don't let git:// through, or more commonly hosted
> servers that don't do the git protocol at all).
>
> Which hopefully clarifies the issue a bit.
Thanks for explanation. Unfortunately I can't use git:// with "git
pull" (at least in git-1.3.2). First it does some traffic, that
suddenly stops - I guess the server starts doing *something*, perhaps
preparing the update for me or whatnot. After a pretty long while it
sends some more data but in the meanwhile my ADSL router dropped the
NAT entry and git sits on my side waiting for data forever. Recently I
tried the same on a system with direct Inet connection and that worked
just fine.
I suggest adding SO_KEEPALIVE option on the git socket.
Goo
^ permalink raw reply
* Re: Security problem
From: Linus Torvalds @ 2006-06-16 6:27 UTC (permalink / raw)
To: Alexander Litvinov; +Cc: Junio C Hamano, git
In-Reply-To: <200606161237.21997.lan@academsoft.ru>
On Fri, 16 Jun 2006, Alexander Litvinov wrote:
>
> > Well, they may not be "safe" - you just need to work a _lot_ harder to
> > corrupt a pack-file in any interesting manner. And again, git-fsck-objects
> > would pick up any such thing going on.
>
> As it shown in pack-objects.c, each object have stored sha1, almost the same
> as file rename.
Yes and no.
The index file has the stored sha1 (and in that sense you can do almost
the same thing as a file rename by just modifying the index file).
But when we actually transfer a pack over from one place to another (ie a
clone or a push), we don't even transfer the index file. Instead, the
index file gets re-generated at the other end.
That's pretty much an on-going theme in most of git - trying to avoid
having metadata, if that can instead of calculated directly.
So again, a "rsync" or a "http" thing that just gets the index and
pack-files directly _as_files_, will actually also download a corrupt
file. The git native protocol is much harder to fool.
git-fsck-objects actually verifies the pack-files and index files in
several ways:
- both the pack-file and the index-file actually contain a SHA1 checksum
of themselves, so any accidental corruption will be picked up (but if
somebody is able to get at the filesystem, they can obviously
re-calculate the SHA1 and update the checksum too)
- the index file also contains the SHA-1 of the pack-file (and that is
then part of the checksum of the index file), again to avoid accidental
corruption or mixing of index and pack-files.
- fsck checks all of these internal SHA-1 checksums, and verifies basic
information (ie number of objects must match etc)
- each object in the index file is unpacked, and its SHA-1 is
re-calculated and checked against what the index file claimed.
So exactly as with individual objects, the pack-files are actually
verified, and on (native-mode) transfer, the names of individual files are
never actually transferred, rather they are re-calculated from the raw
contents at the receiving end.
The pack-files then have a few additional sanity-checks of their own that
should help pinpoint at least the accidental kind of corruption.
But no, the SHA1 checksums of the pack-files are not checked by normal
operations. That would be deadly - trying to check the SHA1 hash of a
pack-file obviously would involve reading it all in, something normal
operations actually try to avoid (normal ops use the index exactly in
order to only read the parts they need).
Perhaps most importantly, after fsck has checked the SHA-1's of each
individual object, it will also do a full reachability check. That, in
many ways, is even more important than checking that each object name
matches its contents (ie there's no missing history either, and the
"tips" of the repository end up basically validating all the rest).
So again, the thing is set up so that doing a full fsck actually does a
_lot_ of integrity checking.
But in the absense of explicit fsck, we do trust the data, even if the
actual _transfer_ of data will recalculate SHA-1's.
> > - if you corrupt the repository, subsequent clones (or even pulls) from
> > the corrupt repository simply won't work if you use the native
> > protocol, because the native protocol doesn't actually trust anything
> > but the actual contents (so if the contents won't match, then neither
> > will the SHA1 names). So the corruption is pretty strictly limited to
> > the _one_ repository that the attacker had write access to.
>
> As I understand sent pack file will contains actial SHA-1 of objects. And any
> hack will be cleary visible.
No, as mentioned, the actual SHA-1's won't ever be sent, so what happens
is that if the repository on the sending side was hacked, the _sending_
side may never even realize it (since it's not necessarily checking the
SHA-1's), but the receiving side will only ever see the raw data, and as
such, it won't ever even _see_ the "false hidden names", because it will
generate a whole new index that purely depends on the data.
And maybe that's exactly what you meant - yes, the hack will be clearly
visible, because the names will now be the "real" ones. You can't hide
things by using a false name.
> > So there's a pretty fundamental "corruption containment" part there.
> ...
> Situation with evil repo is clear to me: you can turst only to trusted commit
> identified by SHA-1
Yes. Exactly.
And once you have a reason to trust a commit, everything you can reach
from that commit is also trustworthy, assuming it passes fsck. IOW, you
only really need to trust the head(s) in your repository.
> > But yeah, I actually still personally do a fair number of
> > "git-fsck-objects". I've never found anything that way since very early on
> > (and back then, the real problem was rsync getting objects that weren't
> > reachable), but I still do it. It makes me feel happier.
>
> As the result: Always fsck repo after pull/clone !
Well, even better, try to avoid pulling from untrusted sources in the
first place ;)
But yes, fsck is actually fairly fast if you do incremental pulls and
repack your repository. To help you do this, there's two modes to fsck:
there's the "full mode", which goes through _everything_, including
pack-files, and there's the "fsck only lose objects", which is the common
one.
So for example, let's say that you only ever repack your repository
locally when it's been "known good" (in fact, repacking in itself will
generally find almost all of the problems that fsck can find, since a full
repack will obviously do the reachability analysis as part of just the
preparatory work). That means that you only ever need to do the quick
default "light fsck" after a pull, since an incremental pull (with the
native protocol) will have unpacked all the pulled objects.
So "fsck after each pull" is not something we do by default, but if you
keep your repo fairly packed, doing so manually (or by just scripting
things) won't even really slow you down, because it will only ever need to
check incrementally - the stuff you've re-packed it doesn't need to check
(assuming you can now trust your local filesystem).
So git certainly gives you the option to be really anal, and doesn't even
make it needlessly hard or expensive, even with large repositories.
Linus
^ permalink raw reply
* Re: 2.6.17-rc6-mm2
From: Linus Torvalds @ 2006-06-16 6:39 UTC (permalink / raw)
To: Goo GGooo; +Cc: linux-kernel, git
In-Reply-To: <ef5305790606152249n2702873fy7b708d9c47c78470@mail.gmail.com>
On Fri, 16 Jun 2006, Goo GGooo wrote:
>
> Thanks for explanation. Unfortunately I can't use git:// with "git
> pull" (at least in git-1.3.2). First it does some traffic, that
> suddenly stops - I guess the server starts doing *something*, perhaps
> preparing the update for me or whatnot.
Yeah, for a big pull, the server will have to think about the objects it
is going to send you.
> I suggest adding SO_KEEPALIVE option on the git socket.
Actually, the really irritating thing is that we actually generate all
these nice status updates, which just makes pulling and cloning a lot more
comfortable, because you actually see what is going on, and what to
expect.
Except they only work over ssh, where we have a separate channel (for
stderr), and with the native git protocol all that nice status work just
gets flushed to /dev/null :(
Dang. It's literally the most irritating part of the thing: the protocol
itself is exactly the same whether you go over ssh:// or over git://, but
that visual information about what is going on is missing, and it's
surprisingly important from a usability standpoint.
And in your case, the usability downside actually turned into a real
accessibility bug.
Oh, well.
Linus
^ permalink raw reply
* Re: Autoconf/Automake
From: Nikolai Weibull @ 2006-06-16 6:51 UTC (permalink / raw)
To: Johannes Schindelin
Cc: Linus Torvalds, Yann Dirson, Alex Riesen, Pavel Roskin, git
In-Reply-To: <Pine.LNX.4.63.0606160105100.7480@wbgn013.biozentrum.uni-wuerzburg.de>
On 6/16/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> In a project I am stuck in, maven is used. It tries -- of all things -- to
> fix a few shortcomings of ant -- which was supposed to fix shortcomings of
> make! And let's face it. Maven is complicated, slow as a dog lacking all
> four feet, and it still does not do the things I can do in three lines
> with make. It's a complete desaster.
But...it uses XML...how can it not be a panacea?
nikolai
^ permalink raw reply
* Re: Security problem
From: Alexander Litvinov @ 2006-06-16 8:18 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0606152300460.5498@g5.osdl.org>
> So git certainly gives you the option to be really anal, and doesn't even
> make it needlessly hard or expensive, even with large repositories.
Thanks for detailed description. Now I can sleep without any worry about my
repo :-)
^ permalink raw reply
* Re: Autoconf/Automake
From: Jerome Lovy @ 2006-06-16 9:06 UTC (permalink / raw)
To: git
In-Reply-To: <20060615174833.GA32247@dspnet.fr.eu.org>
Olivier Galibert wrote:
> On Thu, Jun 15, 2006 at 10:02:10AM -0700, Linus Torvalds wrote:
>
>>These days, there aren't fifteen different versions of UNIX. There's a
>>couple, and it's perfectly ok to actually say "fix your damn system and
>>just install GNU make". It's easier to install GNU make than it is to
>>install autoconf/automake.
>
>
> You should be careful to separate autoconf and automake. Autoconf is
> not so bad, and you can make clean, maintainable Makefile.in and
> config.h.in files with it, because it uses simple substitution. It is
> quite useful to detect available librairies when some are optional,
> and also to lightly[1] ensure that prefix and friends will stay the
> same between make and make install. Also, especially if you hack a
> little bit to alias 'enable' and 'with', you get a sane interface to
> optional feature selection. Oh, and to seperate compilation
> directories too (vpath generation).
I fully agree with Olivier. It seems to me that you don't have to buy
the whole autoconf/automake/libtool stack to leverage the autoconf
functionality. autoconf alone provides the full "autoconfiguration"
framework (running scriptlets and setting substitution variables
accordingly). You still have to write Makefile.in (with statements
looking like: CC=@CC@). Therefore the resulting Makefile is just as
beautiful or as ugly as you wrote the initial Makefile.in: you have full
control over it.
As for dependencies, one shouldn't confuse what is needed on the
autoconfiguration developer's side (in order to build the configure
script from the configure.in file) and what is needed on the installer's
side to run the configure script and process the generated makefile. The
former needs the autoconf package which itself relies on GNU m4. The
latter merely needs a decently compatible Bourne shell and a decently
compatible make.
On the other hand, what you get with automake is a fully automatically
generated makefile, with make targets conforming to the GNU standards.
But then you fully loose control over the Makefile: you don't write the
Makefile.in anymore (automake does it for you) but rather the terce
Makefile.am. In this respect, automake is like imake: you write few
lines of (i)makefile, but then you cannot complain if you don't
understand what comes in the generated makefile ;-) .
Jérôme Lovy
^ permalink raw reply
* Just out I think, yes. Be delighted with
From: Major @ 2006-06-16 10:26 UTC (permalink / raw)
To: glenn
Hello my friend!
Make your girlfriend or wife speechless with increased hardness, richer orgsms and more power in bed
Get everything you need delivered to your door low-cost and fast.
Largest and most recognized brands are working to make you 100% happy with this stuff.
All you need is here: http://www.extremeci.com
We thank you for being interested in our products
^ permalink raw reply
* Get the freshest Now you have chance to do it Delight in
From: Brendan @ 2006-06-16 10:31 UTC (permalink / raw)
To: glenda
Dear member.
Rock hard manhood, multiple explosions and several times more semen volume
Order now and benefit from lowest costs and convenient shipment
Hot deals on stuff produced by well-known brands from worldwide.
Find what you need: http://www.extremeci.com
The prices are really low and the quality it truly very high!
^ permalink raw reply
* Re: [BUG] stgit branch renaming into new dir crashes
From: Catalin Marinas @ 2006-06-16 12:06 UTC (permalink / raw)
To: Yann Dirson; +Cc: GIT list
In-Reply-To: <20060613214053.GD7766@nowhere.earth>
On 13/06/06, Yann Dirson <ydirson@altern.org> wrote:
> When trying to rename a branch to a name including a slash, there is
> no explicit creation of leading dirs, and stgit crashes:
>
> $ stg branch -r multitag dev/multitag
> Traceback (most recent call last):
[...]
What version of StGIT are you using? It seems to be OK with 0.10.
--
Catalin
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox