* Re: Git is not scalable with too many refs/*
From: Jens Lehmann @ 2011-09-09 15:59 UTC (permalink / raw)
To: Martin Fick; +Cc: git
In-Reply-To: <1315529522448-6774328.post@n2.nabble.com>
Am 09.09.2011 02:52, schrieb Martin Fick:
> An update, I bisected it down to this commit:
>
> 88a21979c5717e3f37b9691e90b6dbf2b94c751a
>
> fetch/pull: recurse into submodules when necessary
>
> Since this can be disabled with the --no-recurse-submodules switch, I tried
> that and indeed, even with the latest 1.7.7rc it becomes fast (~8mins)
> again. The strange part about this is that the repository does not have any
> submodules. Anyway, I hope that this can be useful to others since it is a
> workaround which speed things up enormously. Let me know if you have any
> other tests that you want me to perform,
Thanks for nailing that one down. I'm currently looking into bringing back
decent performance here.
^ permalink raw reply
* Re: RFD: leveraging GitHub's asciidoc rendering for our Documentation/
From: Michael Haggerty @ 2011-09-09 15:58 UTC (permalink / raw)
To: Scott Chacon; +Cc: Michael J Gruber, Git Mailing List
In-Reply-To: <CAP2yMaJ+UBHZp0U=QWOZbnbbOuq8vNoJODWtxZPb1sKXTC_UEg@mail.gmail.com>
On 09/09/2011 05:45 PM, Scott Chacon wrote:
> On Fri, Sep 9, 2011 at 7:34 AM, Michael J Gruber
> <git@drmicha.warpmail.net> wrote:
>> which has all the renaming (*.txt -> *.asciidoc) and Makefile and script
>> changes, but is missing some include changes (because include breaks
>> anyway, see below).
>
> I can change this so we can render .asc if that's less ugly. I've
> been meaning to do this for a while, but I don't think I ever
> incorporated it.
What about letting the project set a gitattribute that tells github how
to render particular files? It would not require files to be renamed,
and it would be more flexible.
OTOH it would not be possible to apply gitattributes (or file renamings)
to old revisions, so the history would continue to be rendered naively.
But here's an additional idea: github could provide web access to the
equivalent of $GIT_DIR/info/attributes (a project-wide .gitattributes
file), which would allow the rendering of files in historical revisions
to be customized and would also allow github rendering behavior to be
defined even in projects that do not want github-specific tags in the
.gitattributes files in their project.
Michael
--
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
^ permalink raw reply
* Fwd: RFD: leveraging GitHub's asciidoc rendering for our Documentation/
From: Sitaram Chamarty @ 2011-09-09 15:55 UTC (permalink / raw)
To: Git Mailing List
In-Reply-To: <CAMK1S_hOY-riZnHAZhC32UAA0BYF1YyXUPujj_jFUEjcYC_4ZA@mail.gmail.com>
forgot to copy the list; sorry...
---------- Forwarded message ----------
From: Sitaram Chamarty <sitaramc@gmail.com>
Date: Fri, Sep 9, 2011 at 9:24 PM
Subject: Re: RFD: leveraging GitHub's asciidoc rendering for our Documentation/
To: Michael J Gruber <git@drmicha.warpmail.net>
On Fri, Sep 9, 2011 at 8:04 PM, Michael J Gruber
<git@drmicha.warpmail.net> wrote:
> Hi there,
>
> I've been looking more to GitHub lately and was wondering whether it is
> worth to leverage their automatic asciidoc rendering for our asciidoc
> files. I have put up a test tree at
>
> https://github.com/gitigit/git/tree/githubtest
>
> which has all the renaming (*.txt -> *.asciidoc) and Makefile and script
> changes, but is missing some include changes (because include breaks
> anyway, see below).
>
> The simple renaming already gives a rendered display of blobs for
> simpler asciidoc files like release notes
>
> https://github.com/gitigit/git/blob/githubtest/Documentation/RelNotes/1.7.7.asciidoc
>
> and api documentation
>
> https://github.com/gitigit/git/blob/githubtest/Documentation/technical/api-credentials.asciidoc
>
> For the man pages, there are several problems as can be seen here:
>
> https://github.com/gitigit/git/blob/githubtest/Documentation/git-blame.asciidoc
>
> Our own customisation is not loaded (of course) so that, e.g., the
> linkgit macro does not work; and the include statement makes GitHub's
> parser unhappy and choke.
maybe github will consider supporting linkgit? 3 letters are common anyway :)
> Does anybody feel this is worth pursuing?
For a long time, I relied on github's rendering for all of my
(gitolite) documentation. Eventually I realised it is too slow to
render. More importantly, the whole github "presence" was extraneous
to the manpage, and often distracted my readers.
Eventually I started pre-rendering my documentation to HTML myself and
pushing it to a branch called "gh-pages". Contrast the visual appeal
of the github-rendered page [1] versus the pre-rendered page [2]. (I
admit I do have some very minimal CSS in my version but that's another
plus point for pre-rendering)
[1]: https://github.com/sitaramc/gitolite/blob/pu/README.mkd
[2]: http://sitaramc.github.com/gitolite/README.html
--
Sitaram
^ permalink raw reply
* Re: Git is not scalable with too many refs/*
From: Michael Haggerty @ 2011-09-09 15:51 UTC (permalink / raw)
To: Martin Fick; +Cc: git
In-Reply-To: <4E6A19AD.80100@alum.mit.edu>
I have answered some of my own questions:
On 09/09/2011 03:50 PM, Michael Haggerty wrote:
> 3. Try using the --no-replace-objects option (I assume that it can be
> used like "git --no-replace-objects fetch ..."). In my case this option
> made a dramatic improvement in the runtimes.
This does not seem to help much.
> 4. Try a test using a repository generated something like the test
> script that I posted in [1]. If it also gives pathologically bad
> performance, then it can serve as a test case to use while we debug the
> problem.
Yes, a simple test repo like that created by the script is enough to
reproduce the problem. The slowdown becomes very obvious after only a
few hundred references.
Curiously, "git clone" is very fast under the same circumstances that
"git fetch" is excruciatingly slow.
According to strace, git seems to be repopulating the ref cache after
each new ref is created (it walks through the whole refs subdirectory
and reads every file). Apparently the ref cache is being discarded
completely whenever a ref is added (which can and should be fixed) and
then being reloaded for some reason (though single refs can be inspected
much faster without reading the cache). This situation should be
improved by the hierarchical refcache changes that I'm working on plus
smarter updating (rather than discarding) of the cache when a new
reference is created.
Some earlier speculation in this thread was that that slowdowns might be
caused by "pessimal" ordering of revisions in the walker queue. But my
test repository shards the references in such a way that the lexical
order of the refnames does not correspond to the topological order of
the commits. So that can't be the whole story.
Michael
--
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
^ permalink raw reply
* Re: RFD: leveraging GitHub's asciidoc rendering for our Documentation/
From: Scott Chacon @ 2011-09-09 15:45 UTC (permalink / raw)
To: Michael J Gruber; +Cc: Git Mailing List
In-Reply-To: <4E6A23DB.1040606@drmicha.warpmail.net>
Hey,
On Fri, Sep 9, 2011 at 7:34 AM, Michael J Gruber
<git@drmicha.warpmail.net> wrote:
> which has all the renaming (*.txt -> *.asciidoc) and Makefile and script
> changes, but is missing some include changes (because include breaks
> anyway, see below).
I can change this so we can render .asc if that's less ugly. I've
been meaning to do this for a while, but I don't think I ever
incorporated it.
>
> Our own customisation is not loaded (of course) so that, e.g., the
> linkgit macro does not work; and the include statement makes GitHub's
> parser unhappy and choke.
>
> Does anybody feel this is worth pursuing?
>
> + Nicer blob view
> + Simpler way to judge documentation changes
> - Need to get our asciidoc config in there
> - GitHub's parser neeeds to learn include
If this is interesting to people, I can help out. However, you can
also fix rendering issues, as long as they don't introduce security
issues (which 'include' and macros can do) by fixing it here:
https://github.com/github/markup
That is the rendering engine we use. If you make it work how you want
and it's safe and you send us a pull request, we'll happily take fixes
to the asciidoc renderer.
Scott
^ permalink raw reply
* Re: [PATCH 2/2] push -s: skeleton
From: Jeff King @ 2011-09-09 15:34 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Shawn O. Pearce
In-Reply-To: <7v7h5iwub9.fsf@alter.siamese.dyndns.org>
On Thu, Sep 08, 2011 at 03:19:54PM -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
>
> > Yeah, it is a potential problem, but it just seems wrong to put too much
> > policy work onto the server.
>
> My take on it is somewhat different. The only thing in the end result we
> want to see is that the pushed commits are annotated with GPG signatures
> in the notes tree, and there is no reason for us to cast in stone that
> there has to be any significance in the commit history of the notes tree.
Hmm. Is order really irrelevant? If you push a commit to master, moving
it from X to Y, then push-rewind it back to X, then push a new commit Z,
how do I cryptographically determine the correct final state of master?
I'll see two push-certs, one going X..Y and one X..Z. They'll have
timestamps, which can be used for ordering. But what if the first and
third actions above are done by different people. Now you're trusting
that their clocks are synced to order them properly.
Note that simply keeping an unsigned but ordered notes history doesn't
solve this problem, either; you'd probably want a parent pointer in your
push cert saying "and this is the previous push cert I am based on".
And maybe this is a use case we don't care about. Maybe it's enough for
the push-cert to say "at some point in time, I thought it was a good
idea to push these commits into master; signed, me".
But I'm not really clear on exactly what the security goals are. The
series you sent looks interesting, but I haven't seen the verification
side of these signatures. What are they going to be used for? What
guarantees are we attempting to provide? For that matter, what is our
threat model? What are attackers capable of?
> In a busy hosting site that has many branches being pushed simultaneously,
> it is entirely plausible that the server side may just want to store each
> received push certificate in a new flat file in a filesystem, and have
> asynchronous process sweep the new certificates to update the notes tree,
> possibly creating a single notes tree commit that records updates by
> multiple pushes, for performance purposes, in its implementation of
> record_signed_push() in receive-pack.
OK, I see. It is not "the server can do whatever it likes with the
information" as much as "the server can do whatever it likes, but at the
very least should eventually create a notes tree of a given form".
-Peff
^ permalink raw reply
* Re: [PATCH 0/6] Improved infrastructure for refname normalization
From: Michael Haggerty @ 2011-09-09 15:33 UTC (permalink / raw)
To: gitzilla; +Cc: git, Junio C Hamano, cmn
In-Reply-To: <4E6A1D7D.6050602@gmail.com>
On 09/09/2011 04:06 PM, A Large Angry SCM wrote:
> On 09/09/2011 07:46 AM, Michael Haggerty wrote:
>> As a prerequisite to storing references caches hierarchically (itself
>> needed for performance reasons), here is a patch series to help us get
>> refname normalization under control.
>>
>> The problem is that some UI accepts unnormalized reference names (like
>> "/foo/bar" or "foo///bar" instead of "foo/bar") and passes them on to
>> library routines without normalizing them. The library, on the other
>> hand, assumes that the refnames are normalized. Sometimes (mostly in
>> the case of loose references) unnormalized refnames happen to work,
>> but in other cases (like packed references or when looking up refnames
>> in the cache) they silently fail. Given that refnames are sometimes
>> treated as path names, there is a chance that some security-relevant
>> bugs are lurking in this area, if not in git proper then in scripts
>> that interact with git.
>
> Why can't the library do the normalization instead of expecting every
> other component that deals with reference names having to do it for the
> library?
The library could do the normalization, but
1. It would probably cost a lot of redundant checks as reference names
pass in and out of the library and back in again
2. Normalization requires copying or overwriting the incoming string, so
each time a refname crosses the library perimeter there might have to be
an extra memory allocation with the associated headaches of dealing with
the ownership of the memory.
3. The library doesn't encapsulate all uses of reference names; for
example, for_each_ref() invokes a callback function with the refname as
an argument. The callback function is free to do a strcmp() of the
refname (normalized by the library) with some arbitrary string that it
got from the command line. Either the caller has to do the
normalization itself (i.e., outside of the library) or the library has
to learn how to do every possible filtering operation with refnames.
>> * Forbid ".lock" at the end of any refname component, as directories
>> with such names can conflict with attempts to create lock files for
>> other refnames.
>
> I find this overly restrictive. If you need to create a lock based on a
> reference name or component, use a name for the lock object that starts
> with one of the characters that reference names or components are
> already forbidden from starting with.
I agree; this is unpleasantly restrictive.
But please remember that refnames already cannot end in ".lock"
("foo/bar.lock" is already forbidden; this change also prohibits
"foo.lock/bar").
However, your suggested solution would cause problems if two versions of
git are running on the same machine. An old version of git would not
know to respect the new version's lock files. ISTM that this would be
too dangerous. Suggestions welcome.
Michael
--
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
^ permalink raw reply
* Re: [PATCH 2/2] push -s: skeleton
From: Jeff King @ 2011-09-09 15:22 UTC (permalink / raw)
To: Shawn Pearce; +Cc: Junio C Hamano, git
In-Reply-To: <CAJo=hJsQvRN3Z0xJg9q37Km1g_1qUdJKNQ6n8=a9mv3YjugyVw@mail.gmail.com>
On Thu, Sep 08, 2011 at 03:07:25PM -0700, Shawn O. Pearce wrote:
> A notes tree per ref is ugly when you have a lot of branches. Its a
> problem when you merge commits from say "maint" over to "master" 2
> days after they were initially pushed. Ideally the notes tree for
> master also includes what you brought over.
I agree you can end up with a lot of refs if you have a lot of branches,
and that may get a bit unwieldy. I don't see how the merge thing is a
problem, though. If you do the merge at a client, then those commits
will hit master by push, and you'll get entries in the cert tree for
master. If you do the merge locally, then there's not going to be a
signature on those commits hitting the master ref, no matter what the
storage scheme.
> However, maybe it is reasonable for a protocol extension to support
> "auto-notes-merge" on a refs/notes/ ref if the client asks for it in
> the send-pack/receive-pack command stream? Clients could push notes
> and have the server automatically merge the client's pushed commit
> into the notes tree, either by fast-forward or by performing an
> automatic notes merge, with concat being applied to non-identical
> notes on the same SHA-1. This would allow the client to prepare his
> local certificate, and push that notes tree, while still working
> around the race on the server side.
>
> If the server doesn't support the protocol extension, the client can
> still push his signed notes, he just may run into a race with another
> concurrent user pushing into the same repository. Which then means
> that an upgrade of the server is really only important/necessary if
> you have "central repository" model that a lot of users push into. If
> you use the traditional workflow that GitHub encourages of per-user
> repositories, you would never have a race on the server, and wouldn't
> need the upgraded server binary with "auto-notes-merge".
I like this approach, as the protocol extension provides the minimal
building block that can be used to implement this, or any other
notes-related scheme.
-Peff
^ permalink raw reply
* Re: git repository size / compression
From: neubyr @ 2011-09-09 15:09 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Carlos Martín Nieto, git, pjweisberg
In-Reply-To: <m339g5u5pm.fsf@localhost.localdomain>
2011/9/9 Jakub Narebski <jnareb@gmail.com>:
> neubyr <neubyr@gmail.com> writes:
>> On Fri, Sep 9, 2011 at 3:23 AM, Carlos Martín Nieto <cmn@elego.de> wrote:
>> > On Thu, 2011-09-08 at 21:37 -0500, neubyr wrote:
>
>>>> I have a test git repository with just two files in it. One of the
>>>> file in it has a set of two lines that is repeated n times.
>>>> e.g.:
>>>> {{{
>>>> $ for i in {1..5}; do cat ./lexico.txt>> lexico1.txt && cat
>>>> ./lexico.txt>> lexico1.txt && mv ./lexico1.txt ./lexico.txt; done
>>>> }}}
>>>>
>>>
>>> So you've just created some data that can be compressed quite
>>> efficiently.
>>>
>>>> I ran above command few times and performed commit after each run. Now
>>>> disk usage of this repository directory is mentioned below. The 419M
>>>> is working directory size and 2.7M is git repository/database size.
>>>>
>>>> {{{
>>>> $ du -h -d 1 .
>>>> 2.7M ./.git
>>>> 419M .
>>>>
>>>> }}}
>
> Have you tried the same but with
>
> $ git gc --prune=now
>
> before running `du`?
>
Nope, I hadn't run git gc before. Here are du results after running
git gc command. That's about 55% less space now.. Great!
{{{
$ du -d 1 -h
924K ./.git
417M .
}}}
>>>> Is it because of the compression performed by git before storing data
>>>> (or before sending commit)??
>>>
>>> Yes. Git stores its objects (the commit, the snapshot of the files,
>>> etc.) compressed. When these objects are stored in a pack, the size can
>>> be further reduced by storing some objects as deltas which describe the
>>> difference between itself and some other object in the object-db.
>>
>> Does git store deltas for some files? I thought it uses snapshots
>> (exact copy of staged files) only.
>
> When creating packfile from loose objects (e.g. via `git gc`), it
> does perform delta compression.
>
> --
> Jakub Narębski
>
thank you everyone for explaining in detail..
--
neuby.r
^ permalink raw reply
* Re: git repository size / compression
From: neubyr @ 2011-09-09 15:07 UTC (permalink / raw)
To: Carlos Martín Nieto; +Cc: git
In-Reply-To: <1315578547.4377.2.camel@centaur.lab.cmartin.tk>
On Fri, Sep 9, 2011 at 9:28 AM, Carlos Martín Nieto <cmn@elego.de> wrote:
> On Fri, 2011-09-09 at 09:04 -0500, neubyr wrote:
>> On Fri, Sep 9, 2011 at 3:23 AM, Carlos Martín Nieto <cmn@elego.de> wrote:
>> > On Thu, 2011-09-08 at 21:37 -0500, neubyr wrote:
>> >> I have a test git repository with just two files in it. One of the
>> >> file in it has a set of two lines that is repeated n times.
>> >> e.g.:
>> >> {{{
>> >> $ for i in {1..5}; do cat ./lexico.txt >> lexico1.txt && cat
>> >> ./lexico.txt >> lexico1.txt && mv ./lexico1.txt ./lexico.txt; done
>> >> }}}
>> >>
>> >
>> > So you've just created some data that can be compressed quite
>> > efficiently.
>> >
>> >> I ran above command few times and performed commit after each run. Now
>> >> disk usage of this repository directory is mentioned below. The 419M
>> >> is working directory size and 2.7M is git repository/database size.
>> >>
>> >> {{{
>> >> $ du -h -d 1 .
>> >> 2.7M ./.git
>> >> 419M .
>> >>
>> >> }}}
>> >>
>> >> Is it because of the compression performed by git before storing data
>> >> (or before sending commit)??
>> >>
>> >
>> > Yes. Git stores its objects (the commit, the snapshot of the files,
>> > etc.) compressed. When these objects are stored in a pack, the size can
>> > be further reduced by storing some objects as deltas which describe the
>> > difference between itself and some other object in the object-db.
>> >
>>
>> Does git store deltas for some files? I thought it uses snapshots
>> (exact copy of staged files) only.
>
> Yes and no. The data model for git is to always store snapshots, and it
> always expects to have the full files available. In a packfile, however,
> in order to save space, some objects are stored as deltas to other
> objects in the same file.
>
> http://progit.org/book/ch9-4.html
>
Excellent.. That explains compression and deltas really well. Thanks again..
--
neuby.r
^ permalink raw reply
* Re: git repository size / compression
From: Jakub Narebski @ 2011-09-09 14:54 UTC (permalink / raw)
To: neubyr; +Cc: Carlos Martín Nieto, git
In-Reply-To: <CALFxCvxmPN_O_3xpkrGUYtdkVfz5nr7eaucMrAYQ3uvi820FBg@mail.gmail.com>
neubyr <neubyr@gmail.com> writes:
> On Fri, Sep 9, 2011 at 3:23 AM, Carlos Martín Nieto <cmn@elego.de> wrote:
> > On Thu, 2011-09-08 at 21:37 -0500, neubyr wrote:
>>> I have a test git repository with just two files in it. One of the
>>> file in it has a set of two lines that is repeated n times.
>>> e.g.:
>>> {{{
>>> $ for i in {1..5}; do cat ./lexico.txt>> lexico1.txt && cat
>>> ./lexico.txt>> lexico1.txt && mv ./lexico1.txt ./lexico.txt; done
>>> }}}
>>>
>>
>> So you've just created some data that can be compressed quite
>> efficiently.
>>
>>> I ran above command few times and performed commit after each run. Now
>>> disk usage of this repository directory is mentioned below. The 419M
>>> is working directory size and 2.7M is git repository/database size.
>>>
>>> {{{
>>> $ du -h -d 1 .
>>> 2.7M ./.git
>>> 419M .
>>>
>>> }}}
Have you tried the same but with
$ git gc --prune=now
before running `du`?
>>> Is it because of the compression performed by git before storing data
>>> (or before sending commit)??
>>
>> Yes. Git stores its objects (the commit, the snapshot of the files,
>> etc.) compressed. When these objects are stored in a pack, the size can
>> be further reduced by storing some objects as deltas which describe the
>> difference between itself and some other object in the object-db.
>
> Does git store deltas for some files? I thought it uses snapshots
> (exact copy of staged files) only.
When creating packfile from loose objects (e.g. via `git gc`), it
does perform delta compression.
--
Jakub Narębski
^ permalink raw reply
* RFD: leveraging GitHub's asciidoc rendering for our Documentation/
From: Michael J Gruber @ 2011-09-09 14:34 UTC (permalink / raw)
To: Git Mailing List
Hi there,
I've been looking more to GitHub lately and was wondering whether it is
worth to leverage their automatic asciidoc rendering for our asciidoc
files. I have put up a test tree at
https://github.com/gitigit/git/tree/githubtest
which has all the renaming (*.txt -> *.asciidoc) and Makefile and script
changes, but is missing some include changes (because include breaks
anyway, see below).
The simple renaming already gives a rendered display of blobs for
simpler asciidoc files like release notes
https://github.com/gitigit/git/blob/githubtest/Documentation/RelNotes/1.7.7.asciidoc
and api documentation
https://github.com/gitigit/git/blob/githubtest/Documentation/technical/api-credentials.asciidoc
For the man pages, there are several problems as can be seen here:
https://github.com/gitigit/git/blob/githubtest/Documentation/git-blame.asciidoc
Our own customisation is not loaded (of course) so that, e.g., the
linkgit macro does not work; and the include statement makes GitHub's
parser unhappy and choke.
Does anybody feel this is worth pursuing?
+ Nicer blob view
+ Simpler way to judge documentation changes
- Need to get our asciidoc config in there
- GitHub's parser neeeds to learn include
Michael
^ permalink raw reply
* Re: git repository size / compression
From: Carlos Martín Nieto @ 2011-09-09 14:28 UTC (permalink / raw)
To: neubyr; +Cc: git
In-Reply-To: <CALFxCvxmPN_O_3xpkrGUYtdkVfz5nr7eaucMrAYQ3uvi820FBg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2895 bytes --]
On Fri, 2011-09-09 at 09:04 -0500, neubyr wrote:
> On Fri, Sep 9, 2011 at 3:23 AM, Carlos Martín Nieto <cmn@elego.de> wrote:
> > On Thu, 2011-09-08 at 21:37 -0500, neubyr wrote:
> >> I have a test git repository with just two files in it. One of the
> >> file in it has a set of two lines that is repeated n times.
> >> e.g.:
> >> {{{
> >> $ for i in {1..5}; do cat ./lexico.txt >> lexico1.txt && cat
> >> ./lexico.txt >> lexico1.txt && mv ./lexico1.txt ./lexico.txt; done
> >> }}}
> >>
> >
> > So you've just created some data that can be compressed quite
> > efficiently.
> >
> >> I ran above command few times and performed commit after each run. Now
> >> disk usage of this repository directory is mentioned below. The 419M
> >> is working directory size and 2.7M is git repository/database size.
> >>
> >> {{{
> >> $ du -h -d 1 .
> >> 2.7M ./.git
> >> 419M .
> >>
> >> }}}
> >>
> >> Is it because of the compression performed by git before storing data
> >> (or before sending commit)??
> >>
> >
> > Yes. Git stores its objects (the commit, the snapshot of the files,
> > etc.) compressed. When these objects are stored in a pack, the size can
> > be further reduced by storing some objects as deltas which describe the
> > difference between itself and some other object in the object-db.
> >
>
> Does git store deltas for some files? I thought it uses snapshots
> (exact copy of staged files) only.
Yes and no. The data model for git is to always store snapshots, and it
always expects to have the full files available. In a packfile, however,
in order to save space, some objects are stored as deltas to other
objects in the same file.
http://progit.org/book/ch9-4.html
>
>
> >> Following were results with subversion:
> >>
> >> Subversion client (redundant(?) copy exists in .svn/text-base/
> >> directory, hence double size in client):
> >> {{{
> >> $ du -h -d 1
> >> 416M ./.svn
> >> 832M .
> >> }}}
> >
> > Subversion stores the "pristines" (which is the status of the files in
> > the latest revision) inside the .svn directory. I wouldn't call this
> > copy redundant, though, as it allows you to run diff locally. The
> > pristines are stored uncompressed, which is why you half of the space is
> > taken up by the .svn directory.
> >
> >>
> >> Subversion repo/server:
> >> {{{
> >> $ du -h -d 1
> >> 12K ./conf
> >> 1.2M ./db
> >> 36K ./hooks
> >> 8.0K ./locks
> >> 1.2M .
> >> }}}
> >
> > I don't know how the repository is stored in Subversion, but it may also
> > be compressed. You may be able to reduced your git repository size by
> > (re)generating packs with 'git repack' and doing some cleanups with 'git
> > gc', but the repository size is not often a concern.
> >
> > cmn
> >
> >
> >
>
> that's helpful. thanks.
>
> --
> neuby.r
>
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply
* Re: git repository size / compression
From: Sverre Rabbelier @ 2011-09-09 14:25 UTC (permalink / raw)
To: neubyr; +Cc: Carlos Martín Nieto, git
In-Reply-To: <CALFxCvxmPN_O_3xpkrGUYtdkVfz5nr7eaucMrAYQ3uvi820FBg@mail.gmail.com>
Heya,
On Fri, Sep 9, 2011 at 16:04, neubyr <neubyr@gmail.com> wrote:
> Does git store deltas for some files? I thought it uses snapshots
> (exact copy of staged files) only.
In packs, yes, it will try to delta objects as efficient as possible.
--
Cheers,
Sverre Rabbelier
^ permalink raw reply
* Re: t0300-credentials: poll failed: invalid argument
From: Thomas Rast @ 2011-09-09 14:13 UTC (permalink / raw)
To: Jeff King; +Cc: Brian Gernhardt, Git List
In-Reply-To: <20110829174309.GA11524@sigill.intra.peff.net>
Jeff King wrote:
> On Mon, Aug 29, 2011 at 01:28:05PM -0400, Brian Gernhardt wrote:
>
> > > Ugh, sorry, this is my fault. The check_expiration() function can return
> > > a totally bogus value before we actually get any credentials.
> > >
> > > Does this patch fix it for you?
> >
> > Yes it does! Surprisingly enough, non-bogus parameters keeps poll
> > from erroring with EINVAL. Funny that. ;-)
>
> Great. I'm working on a few more patches on top of that topic, so I'll
> add it to my list to send out in the next day or so.
I'm still seeing this with current pu (from repo.or.cz), but only on
OS X
$ uname -a
Darwin mackeller.inf.ethz.ch 11.1.0 Darwin Kernel Version 11.1.0: Tue Jul 26 16:07:11 PDT 2011; root:xnu-1699.22.81~1/RELEASE_X86_64 x86_64
Where "this" is:
--- expect-stderr 2011-09-09 14:12:13.000000000 +0000
+++ stderr 2011-09-09 14:12:13.000000000 +0000
@@ -1,2 +1,3 @@
askpass: Username:
askpass: Password:
+fatal: poll failed: Invalid argument
for each of the tests 15--19. Is it supposed to be fixed?
I don't have time to look into it without knowing what to search for,
but if you want me to test anything on that OS X just ask.
--
Thomas Rast
trast@{inf,student}.ethz.ch
^ permalink raw reply
* Re: [PATCH 0/6] Improved infrastructure for refname normalization
From: A Large Angry SCM @ 2011-09-09 14:06 UTC (permalink / raw)
To: Michael Haggerty; +Cc: git, Junio C Hamano, cmn
In-Reply-To: <1315568778-3592-1-git-send-email-mhagger@alum.mit.edu>
On 09/09/2011 07:46 AM, Michael Haggerty wrote:
> As a prerequisite to storing references caches hierarchically (itself
> needed for performance reasons), here is a patch series to help us get
> refname normalization under control.
>
> The problem is that some UI accepts unnormalized reference names (like
> "/foo/bar" or "foo///bar" instead of "foo/bar") and passes them on to
> library routines without normalizing them. The library, on the other
> hand, assumes that the refnames are normalized. Sometimes (mostly in
> the case of loose references) unnormalized refnames happen to work,
> but in other cases (like packed references or when looking up refnames
> in the cache) they silently fail. Given that refnames are sometimes
> treated as path names, there is a chance that some security-relevant
> bugs are lurking in this area, if not in git proper then in scripts
> that interact with git.
Why can't the library do the normalization instead of expecting every
other component that deals with reference names having to do it for the
library?
[...]
>
> * Forbid ".lock" at the end of any refname component, as directories
> with such names can conflict with attempts to create lock files for
> other refnames.
I find this overly restrictive. If you need to create a lock based on a
reference name or component, use a name for the lock object that starts
with one of the characters that reference names or components are
already forbidden from starting with.
Gitzilla
^ permalink raw reply
* Re: git repository size / compression
From: neubyr @ 2011-09-09 14:04 UTC (permalink / raw)
To: Carlos Martín Nieto; +Cc: git
In-Reply-To: <1315556595.2019.11.camel@bee.lab.cmartin.tk>
On Fri, Sep 9, 2011 at 3:23 AM, Carlos Martín Nieto <cmn@elego.de> wrote:
> On Thu, 2011-09-08 at 21:37 -0500, neubyr wrote:
>> I have a test git repository with just two files in it. One of the
>> file in it has a set of two lines that is repeated n times.
>> e.g.:
>> {{{
>> $ for i in {1..5}; do cat ./lexico.txt >> lexico1.txt && cat
>> ./lexico.txt >> lexico1.txt && mv ./lexico1.txt ./lexico.txt; done
>> }}}
>>
>
> So you've just created some data that can be compressed quite
> efficiently.
>
>> I ran above command few times and performed commit after each run. Now
>> disk usage of this repository directory is mentioned below. The 419M
>> is working directory size and 2.7M is git repository/database size.
>>
>> {{{
>> $ du -h -d 1 .
>> 2.7M ./.git
>> 419M .
>>
>> }}}
>>
>> Is it because of the compression performed by git before storing data
>> (or before sending commit)??
>>
>
> Yes. Git stores its objects (the commit, the snapshot of the files,
> etc.) compressed. When these objects are stored in a pack, the size can
> be further reduced by storing some objects as deltas which describe the
> difference between itself and some other object in the object-db.
>
Does git store deltas for some files? I thought it uses snapshots
(exact copy of staged files) only.
>> Following were results with subversion:
>>
>> Subversion client (redundant(?) copy exists in .svn/text-base/
>> directory, hence double size in client):
>> {{{
>> $ du -h -d 1
>> 416M ./.svn
>> 832M .
>> }}}
>
> Subversion stores the "pristines" (which is the status of the files in
> the latest revision) inside the .svn directory. I wouldn't call this
> copy redundant, though, as it allows you to run diff locally. The
> pristines are stored uncompressed, which is why you half of the space is
> taken up by the .svn directory.
>
>>
>> Subversion repo/server:
>> {{{
>> $ du -h -d 1
>> 12K ./conf
>> 1.2M ./db
>> 36K ./hooks
>> 8.0K ./locks
>> 1.2M .
>> }}}
>
> I don't know how the repository is stored in Subversion, but it may also
> be compressed. You may be able to reduced your git repository size by
> (re)generating packs with 'git repack' and doing some cleanups with 'git
> gc', but the repository size is not often a concern.
>
> cmn
>
>
>
that's helpful. thanks.
--
neuby.r
^ permalink raw reply
* Re: can Git encrypt/decrypt .gpg on push/fetch?
From: Ted Zlatanov @ 2011-09-09 13:52 UTC (permalink / raw)
To: Michael J Gruber; +Cc: Aneesh Bhasin, git
In-Reply-To: <4E6A165D.5010703@drmicha.warpmail.net>
On Fri, 09 Sep 2011 15:36:29 +0200 Michael J Gruber <git@drmicha.warpmail.net> wrote:
MJG> Aneesh Bhasin venit, vidit, dixit 09.09.2011 12:50:
>> Hi Ted,
>>
>>
>> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>
>>>
>>> I need to store some encrypted files in Git but for some clients with
>>> the right GPG keys, decrypt them on checkout (possibly also encrypt them
>>> back on commit, but that's not as important).
>>>
>>> diff doesn't have to work, this is just for convenience. Can Git do
>>> this (matching only .gpg files) or do I need my own command to run after
>>> the checkout/fetch and before commit? It seems pretty out of Git's
>>> scope but perhaps others have done this before.
>>>
>>
>> Have you looked at git hooks (e.g. here : http://progit.org/book/ch7-3.html).
>>
>> You could do the encryption/decryption in pre-commit and post-checkout
>> hooks scripts respectively...
MJG> I'd recommend textconv for diffing and clean/smudge for plaintext
MJG> checkout. That is, there are two convenient versions:
MJG> A) Keep blobs and checkout encrypted
MJG> - Use an editor which can encrypt/decrypt on the fly (e.g. vim)
MJG> - Use "*.gpg diff=gpg" in your attributes and
MJG> [diff "gpg"]
MJG> textconv = gpg -d
MJG> in your config to have cleartext diffs. Use cachetextconv with caution ;)
MJG> B) Keep blobs encrypted, checkout decrypted
MJG> - Use Use "*.gpg filter=gpg" in your attributes and
MJG> [filter "gpg"]
MJG> smudge = gpg -d
MJG> clean = gpg -e -r yourgpgkey
MJG> in your config.
MJG> I use A on a regular basis. B is untested (but patterned after a similar
MJG> gzip filter I use). You may or may not have better results with "gpg -ea".
MJG> On clients without the keys, you can simply leave out the diff or filter
MJG> config resp. set them to "cat".
That's really helpful, thank you Aneesh and Michael. Exactly what I was
hoping to achieve.
Ted
^ permalink raw reply
* Re: Git is not scalable with too many refs/*
From: Michael Haggerty @ 2011-09-09 13:50 UTC (permalink / raw)
To: Martin Fick; +Cc: git
In-Reply-To: <1315511619144-6773496.post@n2.nabble.com>
On 09/08/2011 09:53 PM, Martin Fick wrote:
> Just thought that I should add some numbers to this thread as it seems that
> the later versions of git are worse off by several orders of magnitude on
> this one.
>
> We have a Gerrit repo with just under 100K refs in refs/changes/*. When I
> fetch them all with git 1.7.6 it does not seem to complete. Even after 5
> days, it is just under half way through the ref #s! [...]
I recently reported very slow performance when doing a "git
filter-branch" involving only about 1000 tags, with hints of O(N^3)
scaling [1]. That could certainly explain enormous runtimes for 100k refs.
References are cached in git in a single linked list, so it is easy to
imagine O(N^2) all over the place (which is bad enough for 100k
references). I am working on improving the situation by reorganizing
how the reference cache is stored in memory, but progress is slow.
I'm not sure whether your problem is related. For example, it is not
obvious to me why the commit that you cite (88a21979) would make the
reference problem so dramatically worse.
I suggest the following experiments to characterize the problem:
1. Fetch the references in batches of a few hundred each, and see if
that dramatically decreases the total time.
2. Same as (1), except run "git pack-refs --all --prune" between the
batches. In my experiments, packing references made a dramatic
difference in runtimes.
3. Try using the --no-replace-objects option (I assume that it can be
used like "git --no-replace-objects fetch ..."). In my case this option
made a dramatic improvement in the runtimes.
4. Try a test using a repository generated something like the test
script that I posted in [1]. If it also gives pathologically bad
performance, then it can serve as a test case to use while we debug the
problem.
Yours,
Michael
[1] http://comments.gmane.org/gmane.comp.version-control.git/177103
--
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
^ permalink raw reply
* Re: The imporantance of including http credential caching in 1.7.7
From: Ted Zlatanov @ 2011-09-09 13:31 UTC (permalink / raw)
To: John Szakmeister; +Cc: git, Kyle Neath, Jeff King
In-Reply-To: <CAEBDL5VtVZcmQnj2CH7XzZ0YV_X61gO69-dXriGiYsAqk=NLPg@mail.gmail.com>
On Fri, 9 Sep 2011 06:32:25 -0400 John Szakmeister <john@szakmeister.net> wrote:
JS> [Added back some of the CC's]
JS> Ted: we don't usually cull the CC list on the git mailing list.
Sorry, I followed up via GMane (a NNTP interface to the mailing list).
I'll use `r'eply instead of `f'ollowup.
I actually set
Mail-Copies-To: never
Gmane-Reply-To-List: yes
because I hate getting CC'd on discussions I already follow via GMane.
But that's just my preference :)
JS> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>:
JS> [snip]
MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X
MJG> developers here.
>>
>> Windows doesn't have a standard keychain service, does it?
JS> No, it doesn't, but you can use the wincrypt API which allows you to
JS> at least encrypt the password from the user's login credentials. In
JS> particular, CryptProtectData() and CryptUnprotectData(). That way you
JS> can at least have the password stored encrypted on disk.
I don't think that's sufficient but could be wrong.
Ted
^ permalink raw reply
* Re: The imporantance of including http credential caching in 1.7.7
From: Ted Zlatanov @ 2011-09-09 13:33 UTC (permalink / raw)
To: John Szakmeister; +Cc: kusmabite, git, Kyle Neath, Jeff King
In-Reply-To: <CAEBDL5UymsgsQnE9t121omGkR3Zw9BsFqOinTshxEN4WDcOdeA@mail.gmail.com>
On Fri, 9 Sep 2011 06:54:15 -0400 John Szakmeister <john@szakmeister.net> wrote:
JS> On Fri, Sep 9, 2011 at 6:48 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
JS> [snip]
>> Actually, it seems recent Windows versions does have a credential
>> manager, including an API:
>>
>> http://www.yanzzee.com/2009/09/windows-keychain.html
>> http://msdn.microsoft.com/en-us/library/aa374731(v=VS.85).aspx#credentials_management_functions
JS> Yay! It's about time they grew that feature. :-)
That could work. I hope a Windows developer can take a look. That
leaves VMS credential support... Come on guys!
Ted
^ permalink raw reply
* Re: can Git encrypt/decrypt .gpg on push/fetch?
From: Michael J Gruber @ 2011-09-09 13:36 UTC (permalink / raw)
To: Aneesh Bhasin; +Cc: tzz, git
In-Reply-To: <CAGhXAGSw3y=cjAHXtwycDifoBPr13AkYtLHRRXejRKue0vkz7A@mail.gmail.com>
Aneesh Bhasin venit, vidit, dixit 09.09.2011 12:50:
> Hi Ted,
>
>
> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>
>>
>> I need to store some encrypted files in Git but for some clients with
>> the right GPG keys, decrypt them on checkout (possibly also encrypt them
>> back on commit, but that's not as important).
>>
>> diff doesn't have to work, this is just for convenience. Can Git do
>> this (matching only .gpg files) or do I need my own command to run after
>> the checkout/fetch and before commit? It seems pretty out of Git's
>> scope but perhaps others have done this before.
>>
>
> Have you looked at git hooks (e.g. here : http://progit.org/book/ch7-3.html).
>
> You could do the encryption/decryption in pre-commit and post-checkout
> hooks scripts respectively...
I'd recommend textconv for diffing and clean/smudge for plaintext
checkout. That is, there are two convenient versions:
A) Keep blobs and checkout encrypted
- Use an editor which can encrypt/decrypt on the fly (e.g. vim)
- Use "*.gpg diff=gpg" in your attributes and
[diff "gpg"]
textconv = gpg -d
in your config to have cleartext diffs. Use cachetextconv with caution ;)
B) Keep blobs encrypted, checkout decrypted
- Use Use "*.gpg filter=gpg" in your attributes and
[filter "gpg"]
smudge = gpg -d
clean = gpg -e -r yourgpgkey
in your config.
I use A on a regular basis. B is untested (but patterned after a similar
gzip filter I use). You may or may not have better results with "gpg -ea".
On clients without the keys, you can simply leave out the diff or filter
config resp. set them to "cat".
Michael
^ permalink raw reply
* Re: can Git encrypt/decrypt .gpg on push/fetch?
From: Ted Zlatanov @ 2011-09-09 13:27 UTC (permalink / raw)
To: git
In-Reply-To: <CAGhXAGSw3y=cjAHXtwycDifoBPr13AkYtLHRRXejRKue0vkz7A@mail.gmail.com>
On Fri, 9 Sep 2011 16:20:10 +0530 Aneesh Bhasin <contact.aneesh@gmail.com> wrote:
AB> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>
>>
>> I need to store some encrypted files in Git but for some clients with
>> the right GPG keys, decrypt them on checkout (possibly also encrypt them
>> back on commit, but that's not as important).
>>
>> diff doesn't have to work, this is just for convenience. Can Git do
>> this (matching only .gpg files) or do I need my own command to run after
>> the checkout/fetch and before commit? It seems pretty out of Git's
>> scope but perhaps others have done this before.
>>
AB> Have you looked at git hooks (e.g. here : http://progit.org/book/ch7-3.html).
AB> You could do the encryption/decryption in pre-commit and post-checkout
AB> hooks scripts respectively...
Yes, thank you. I was wondering if there could be further support so
they are checked out in a binary form on the server side if you don't
have the keys but in text form if you do. So for instance "git log -p"
will DTRT on a client with the keys but not on a client without them.
This could require deep Git changes so I'm wondering if it's even
theoretically possible.
Thanks
Ted
^ permalink raw reply
* Re: git-rebase skips automatically no more needed commits
From: Francis Moreau @ 2011-09-09 13:25 UTC (permalink / raw)
To: Martin von Zweigbergk; +Cc: Junio C Hamano, Michael J Gruber, git
In-Reply-To: <alpine.DEB.2.00.1109090900301.12564@debian>
On Fri, Sep 9, 2011 at 3:06 PM, Martin von Zweigbergk
<martin.von.zweigbergk@gmail.com> wrote:
> On Fri, 9 Sep 2011, Francis Moreau wrote:
>
>> On Fri, Sep 9, 2011 at 4:23 AM, Martin von Zweigbergk
>> <martin.von.zweigbergk@gmail.com> wrote:
>>
>> > you are right that
>> > "git rebase --onto master foo~10 foo" could potentially filter out
>> > patches already in foo..master, without calculating patch-ids for all
>> > commits in master..foo for that matter. I think that would make sense
>>
>> you meant master..foo~10, didn't you ?
>
> Yes. (I actually did mean "all commits in master..foo", but "_any_
> commits in master..foo~10" is more clear and exact.)
>
>> please let me know when you submitting your work, I'm interested to see it.
>
> Will do. It's not really that much work, but I'm not sure when I will
> have time for it. If you or anyone else is interested in doing it, you
> are of course welcome.
I'm a simple mortal user ;)
>
>> > necessary, we could have a flag to disable the filtering e.g. when the
>> > user knows that master is part of a completely separate history from
>> > foo.
>>
>> Can't git figure this out itself ? (I'm not saying the switch is useless)
>
> Yes, but this hypothetical flag would only be to speed things up by
> avoid going to the roots to find out that the histories are disjoint.
You're right but in my (mortal) experience, I'm always rebasing
branches onto another one which is not disjoint, so I'm assuming it's
the (very) common case, but I may be wrong. And what I'm assuming is
correct, I think it's make more sense to do the filtering by default
unless the flag tells to do otherwise.
Thanks
--
Francis
^ permalink raw reply
* Re: git-rebase skips automatically no more needed commits
From: Martin von Zweigbergk @ 2011-09-09 13:06 UTC (permalink / raw)
To: Francis Moreau
Cc: Martin von Zweigbergk, Junio C Hamano, Michael J Gruber, git
In-Reply-To: <CAC9WiBhApjEr-NYm9NkyXaZPNCivoJ65c=xx2bg5Li60kfZMTQ@mail.gmail.com>
On Fri, 9 Sep 2011, Francis Moreau wrote:
> On Fri, Sep 9, 2011 at 4:23 AM, Martin von Zweigbergk
> <martin.von.zweigbergk@gmail.com> wrote:
>
> > you are right that
> > "git rebase --onto master foo~10 foo" could potentially filter out
> > patches already in foo..master, without calculating patch-ids for all
> > commits in master..foo for that matter. I think that would make sense
>
> you meant master..foo~10, didn't you ?
Yes. (I actually did mean "all commits in master..foo", but "_any_
commits in master..foo~10" is more clear and exact.)
> please let me know when you submitting your work, I'm interested to see it.
Will do. It's not really that much work, but I'm not sure when I will
have time for it. If you or anyone else is interested in doing it, you
are of course welcome.
> > necessary, we could have a flag to disable the filtering e.g. when the
> > user knows that master is part of a completely separate history from
> > foo.
>
> Can't git figure this out itself ? (I'm not saying the switch is useless)
Yes, but this hypothetical flag would only be to speed things up by
avoid going to the roots to find out that the histories are disjoint.
Martin
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox