* Re: git clone over http
From: Josh England @ 2007-09-04 16:54 UTC (permalink / raw)
To: Robin Rosenberg; +Cc: git
In-Reply-To: <200709021123.04218.robin.rosenberg.lists@dewire.com>
On Sun, 2007-09-02 at 11:23 +0200, Robin Rosenberg wrote:
> git clone (1.5.3) with http is somewhat unreliable. I've noticed if
> can actually give
> me different versions of a branch each time I run it, eventually yielding
> the one I'm expecting. And now this:
I don't see this behavior, but I do see other wierdness cloning via
http. Anytime I clone a repo (any repo) via http from behind the
firewall/proxy at work I never get any branches. 'git branch' only ever
shows 'master'. I can't checkout and work on any other branch. I can
work around the problem by cloning on a box that is more open, but was
wondering if anyone knows how to fix this.
-JE
^ permalink raw reply
* Re: Git's database structure
From: Jeff King @ 2007-09-04 17:09 UTC (permalink / raw)
To: Jon Smirl; +Cc: Andreas Ericsson, Git Mailing List
In-Reply-To: <9e4733910709040919u3d252b91s2785ed4d20086c88@mail.gmail.com>
On Tue, Sep 04, 2007 at 12:19:33PM -0400, Jon Smirl wrote:
> By introducing tree nodes you have blended a specific indexing scheme
> into the data. There are many other ways the path names could be
> indexed hash tables, binary trees, etc.
That is correct. However, given that indexing scheme, many of the common
operations just "fall out" simply and efficiently, without the need to
keep separate indices. So yes, git is geared towards a particular set of
operations.
Your complaint seems to be two-fold:
1. there is an inelegance in the blending of data and indexing. The
problem with changing this is:
a. we are all already using git, and it would require completely
re-vamping the core data structure
b. there is some feeling that the blending is necessary for
performance. Given the difficulty of (a), I think you would
have to provide compelling evidence (i.e., numbers) that a
git-like system based around set theory with separate indices
would perform as well.
2. you want perform some operations to which the hierarchy is not
well-suited. In this case, I think you can get by with the same
solution you have proposed already: indices external to the data
structure (in fact, this is exactly what Google is doing: taking
hierarchical URLs and indexing them in different ways).
Have you taken a look at the pack v4 work by Shawn and Nicolas? It
is an attempt to build such indices at pack time (but keeping the
core git data structure intact).
-Peff
^ permalink raw reply
* Re: [PATCH] Rework strbuf API and semantics.
From: Wincent Colaiuta @ 2007-09-04 17:18 UTC (permalink / raw)
To: Pierre Habouzit; +Cc: Johannes Schindelin, git
In-Reply-To: <20070904161843.GB3381@artemis.corp>
El 4/9/2007, a las 18:18, Pierre Habouzit escribió:
> ooooh, now I'm guilty of not knowing all git APIs very well
> yet :) Indeed,
> this should just be:
>
> void strbuf_grow(struct strbuf *sb, size_t extra) {
> if (sb->len + extra + 1 < sb->len)
> die("you want to use way to much memory");
> ALLOC_GROW(sb->buf, sb->len + extra + 1, sb->alloc);
> }
"too much memory", not "to much memory"
Cheers,
Wincent
^ permalink raw reply
* Re: Git's database structure
From: Julian Phillips @ 2007-09-04 17:19 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910709040823k731f0ffchba1f93bdb4a8373d@mail.gmail.com>
On Tue, 4 Sep 2007, Jon Smirl wrote:
> Let's back up a little bit from "Caclulating tree node". What are the
> elements of git's data structures?
>
> Right now we have an index structure (tree nodes) integrated in to a
> base table. Integrating indexing into the data is not normally done in
> a database. Doing a normalization analysis like this may expose flaws
> in the way the data is structured. Of course we may also decide to
> leave everything the way it is.
>
> What about the special status of a rename? In the current model we
> effectively have three tables.
>
> commit - a set of all SHAs in the commit, previous commit, comment, author, etc
> blob - a file, permissions, etc.
> file names - name, SHA
>
> The file name table is encoded as an index and it has been
> intermingled with the commit table.
>
> Looking at this from a set theory angle brings up the question, do we
> really have three tables and file names are an independent variable
> from the blobs, or should file names be an attribute of the blob?
There isn't a one-to-one mapping of file names to blobs. The blob only
describes the contents of the file. In the extreme case you could have
one blob for every single file in your tree. For example:
# git ls-tree -r HEAD
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c bar/foo
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo2
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo3
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo4
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo5
100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo6
>
> How this gets structured in the db is an independent question about
> how renames get detected on a commit. The current scheme for detecting
> renames by comparing diffs is working fine. The question is, once we
> detect a rename how should it be stored?
>
> Ignoring the performance impacts and looking at the problem from the
> set theory view point, should:
> the pathnames be in their own table with a row for each alias
> the pathnames be stored as an attribute of the blob
>
> Both of these are the same information, we're just looking at how
> things are normalized.
>
>
--
Julian
---
"You shouldn't make my toaster angry."
-- Household security explained in "Johnny Quest"
^ permalink raw reply
* Re: Git's database structure
From: Junio C Hamano @ 2007-09-04 17:21 UTC (permalink / raw)
To: Andreas Ericsson; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <46DD7FE4.1060908@op5.se>
Andreas Ericsson <ae@op5.se> writes:
> Each root tree can only ever belong to a single commit, unless you
> intentionally force git to make completely empty commits. git
> won't complain about this, so long as you don't make two in the
> same second, because it relies more heavily on the DAG than on
> developer sanity.
This actually can happen without even using 'ours' strategy.
If two people independently applied the same patch on their
branches and later their results were merged. And "the same
second" requirement is not even there and not interesting.
There are other things like developer identity, log message, and
their ancestry that would make the resulting commit object
distinct.
> Each root tree can point to multiple sub-trees. The sub-trees can be
> linked to any number of root-trees.
>
> Blobs can be linked to any number of tree objects, or even multiple
> times to the same tree object. This wouldn't be possible if the
> blob objects had their own pathnames stored inside them, so to speak.
More importantly, in git, filenames and modes are not considered
part of "contents", which git tracks. Although it is an
entirely possible and valid alternate design to move that as
part of "blob" to build a system that is different from git,
which Jon seems to be aiming at, the benefit of such a design is
unclear to me, both from theoretical point of view (now blobs
are not about pure contents anymore) nor performance point of
view (Linus's done flat tree object in an early stage of git,
and it was not nice) as other people explained.
^ permalink raw reply
* Re: Git's database structure
From: Junio C Hamano @ 2007-09-04 17:25 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910709040928n6535e49esaf713b2c63ba0831@mail.gmail.com>
"Jon Smirl" <jonsmirl@gmail.com> writes:
> Another way of looking at the problem,
>
> Let's build a full-text index for git. You put a string into the index
> and it returns the SHAs of all the file nodes that contain the string.
> How do I recover the path names of these SHAs?
That question does not make much sense without specifying "which
commit's path you are talking about".
If you want to encode such "contextual information" in addition
to "contents", you could do so, but you essentially need to
record commit + pathname + mode bits + contents as "blob" and
hash that to come up with a name.
^ permalink raw reply
* Re: [PATCH] Add post-merge hook.
From: Junio C Hamano @ 2007-09-04 17:25 UTC (permalink / raw)
To: Josh England; +Cc: git
In-Reply-To: <1188923110.6192.15.camel@beauty>
"Josh England" <jjengla@sandia.gov> writes:
>> Two questions.
>>
>> * Do you want to run the post-merge hook even for a squash
>> merge?
>
> Yes. I'd like to run it at any time that the working tree might be
> updated.
If that is the case, perhaps your hook may want to get a
parameter to tell it what kind of "git-merge" invocation it was?
Squash merge does not even advance the HEAD and is of a very
different nature from a normal merge.
>> - We would want a new test in the test suite for this, to make
>> sure that later changes by others would not break this new
>> feature you would depend upon.
>
> Can do. You want me to resubmit the original patch along with tests or
> submit the tests as a new patch?
I'd like a full resend whenever I reject a patch with a
comment. That way the patch will be easier to review with
context by other people.
Thanks.
^ permalink raw reply
* Re: git clone over http
From: Junio C Hamano @ 2007-09-04 17:29 UTC (permalink / raw)
To: Josh England; +Cc: Robin Rosenberg, git
In-Reply-To: <1188924876.6192.21.camel@beauty>
"Josh England" <jjengla@sandia.gov> writes:
> On Sun, 2007-09-02 at 11:23 +0200, Robin Rosenberg wrote:
>> git clone (1.5.3) with http is somewhat unreliable. I've noticed if
>> can actually give
>> me different versions of a branch each time I run it, eventually yielding
>> the one I'm expecting. And now this:
>
> I don't see this behavior, but I do see other wierdness cloning via
> http. Anytime I clone a repo (any repo) via http from behind the
> firewall/proxy at work I never get any branches. 'git branch' only ever
> shows 'master'. I can't checkout and work on any other branch. I can
> work around the problem by cloning on a box that is more open, but was
> wondering if anyone knows how to fix this.
Well, what does "git branch -r" say?
If you see the remote tracking branches out of your branch (aka
"heads/") namespace, that has been the default for quite some
time since 1.5.0 days.
Perhaps "more open" one has ancient git that dumped remote
tracking branches in your branch namespace, while your "behind
firewall" one has not so ancient git?
^ permalink raw reply
* Re: Git's database structure
From: Jon Smirl @ 2007-09-04 17:30 UTC (permalink / raw)
To: Julian Phillips; +Cc: Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0709041816340.29009@reaper.quantumfyre.co.uk>
On 9/4/07, Julian Phillips <julian@quantumfyre.co.uk> wrote:
> On Tue, 4 Sep 2007, Jon Smirl wrote:
>
> > Let's back up a little bit from "Caclulating tree node". What are the
> > elements of git's data structures?
> >
> > Right now we have an index structure (tree nodes) integrated in to a
> > base table. Integrating indexing into the data is not normally done in
> > a database. Doing a normalization analysis like this may expose flaws
> > in the way the data is structured. Of course we may also decide to
> > leave everything the way it is.
> >
> > What about the special status of a rename? In the current model we
> > effectively have three tables.
> >
> > commit - a set of all SHAs in the commit, previous commit, comment, author, etc
> > blob - a file, permissions, etc.
> > file names - name, SHA
> >
> > The file name table is encoded as an index and it has been
> > intermingled with the commit table.
> >
> > Looking at this from a set theory angle brings up the question, do we
> > really have three tables and file names are an independent variable
> > from the blobs, or should file names be an attribute of the blob?
>
> There isn't a one-to-one mapping of file names to blobs. The blob only
> describes the contents of the file. In the extreme case you could have
> one blob for every single file in your tree. For example:
>
> # git ls-tree -r HEAD
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c bar/foo
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo2
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo3
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo4
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo5
> 100644 blob 05303ef858aeeb01ca40590dd6fe65928096ee6c foo6
Both schemes support aliasing. In the flat scheme you would create a
second blob which contains the file and the aliased path name. When
the blob gets delta'd the second copy of the file will disappear.
I'm not proposing a change to data being stored in git, it is a
proposal to consider the impacts of how this data has been normalized
in the data store.
> > How this gets structured in the db is an independent question about
> > how renames get detected on a commit. The current scheme for detecting
> > renames by comparing diffs is working fine. The question is, once we
> > detect a rename how should it be stored?
> >
> > Ignoring the performance impacts and looking at the problem from the
> > set theory view point, should:
> > the pathnames be in their own table with a row for each alias
> > the pathnames be stored as an attribute of the blob
> >
> > Both of these are the same information, we're just looking at how
> > things are normalized.
> >
> >
>
> --
> Julian
>
> ---
> "You shouldn't make my toaster angry."
> -- Household security explained in "Johnny Quest"
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Calculating tree nodes
From: Junio C Hamano @ 2007-09-04 17:39 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <20070904062629.GZ18160@spearce.org>
"Shawn O. Pearce" <spearce@spearce.org> writes:
> There's nothing stopping us from creating additional indexes.
> ...
> But we can also store the notes alongside the commits in the
> packfile, so that if the data for the commit has been paged in
> by the kernel then the note data is also most likely in memory,
> and if not, is in the read-ahead queue. Clustering the notes
> alongside the commits makes access to them even faster, as we
> don't need to consult an external hash to locate the position.
I would agree with your main thrust "nobody prevents you from
building additional index", but on a tangent, I am skeptical
about adding too much to pack v4. Especially "clustering the
notes" part.
Many operations (like "git log" that is not path limited) do not
even need trees. The current packfile format has commits at the
beginning without any associated trees, and they are stored in
traversal order (modulo delta-base requirements can move base
object earlier), which is geared toward optimizing such a common
case.
Now, hopefully many operations do not need notes either,
although notes themselves can store _anything_ so each of them
could be large and/or each commit could have large number of
them. I suspect clustering notes along with the commit they
annotate would break the locality of access for common case.
^ permalink raw reply
* Re: Git's database structure
From: Jon Smirl @ 2007-09-04 17:44 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List
In-Reply-To: <7vtzqany0z.fsf@gitster.siamese.dyndns.org>
On 9/4/07, Junio C Hamano <gitster@pobox.com> wrote:
> "Jon Smirl" <jonsmirl@gmail.com> writes:
>
> > Another way of looking at the problem,
> >
> > Let's build a full-text index for git. You put a string into the index
> > and it returns the SHAs of all the file nodes that contain the string.
> > How do I recover the path names of these SHAs?
>
> That question does not make much sense without specifying "which
> commit's path you are talking about".
>
> If you want to encode such "contextual information" in addition
> to "contents", you could do so, but you essentially need to
> record commit + pathname + mode bits + contents as "blob" and
> hash that to come up with a name.
I left the details out of the full-text example to make it more
obvious that we can't recover the path names.
Doing this type of analysis may point out that even more fields are
missing from the blob table such as commit id.
The current data store design is not very flexible. Databases solved
the flexibility problem long ago. I'm just wondering if we should
steal some good ideas out of the database world and apply them to git.
Ten years from now we may have 100GB git databases and really wish we
had more flexible ways of querying them.
The reason databases don't encode the fields into the index is that
you can only have a single index on the table if you do that.
Databases do sometimes duplicate the field in both the index and the
table. Databases also have the property that indexes are just a cache
and can be dropped at any time.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: git-svn and a nested branches folder
From: Peter Baumann @ 2007-09-04 17:40 UTC (permalink / raw)
To: Russ Brown; +Cc: git
In-Reply-To: <46DD77F2.3040000@gmail.com>
On Tue, Sep 04, 2007 at 10:21:22AM -0500, Russ Brown wrote:
> David Kastrup wrote:
> > Russ Brown <pickscrape@gmail.com> writes:
> >
> >> David Kastrup wrote:
> >>> Russ Brown <pickscrape@gmail.com> writes:
> >>>
> >>>> I'm having some trouble with using git-svn to fetch a repository, and I
> >>>> think it's because the repository doesn't store branches as a flat list
> >>>> directly under the 'branches' directory.
> >>>>
> >>>> Basically, we have a structure like this:
> >>>>
> >>>> |
> >>>> +-trunk
> >>>> +-tags
> >>>> +-branches
> >>>> + category-a
> >>>> + branch-a
> >>>> + branch-b
> >>>> + category-b
> >>>> + branch-c
> >>>> + branch-d
> >>>>
> >>>> etc. category-a and category-b are simple directories created using svn
> >>>> mkdir. The branches are created using svn cp.
> >>>>
> >>>> It helps us to organise the branches better, but the rationale is
> >>>> besides the point. The problem is that git-svn seems to want to
> >>>> treat category-a and category-b as branches, which isn't right at
> >>>> all. As a result, git-svn seems to skip most (if not all) revisions
> >>>> that occur in these directories and creates a lot of entries in
> >>>> unhandled.log.
> >>> So what did you specify in your .git/config file regarding the svn
> >>> structure?
> >> I specified the 'branches' directory, but that's because earlier in
> >> the life of the repo we did just do the flat branch layout, but
> >> decided to make it more structured once that got unwieldy.
> >
> > Cough, cough. _What_ did you specify in your .git/config file
> > regarding the svn structure? Please quote the section.
> >
>
> Erm, sorry.
>
> [svn-remote "svn"]
> url = svn://svn.<name>.com
> fetch = trunk:refs/remotes/trunk
> branches = branches/*:refs/remotes/*
> tags = tags/*:refs/remotes/tags/*
>
> (URL changed in case it annoys my employers)
>
> I didn't write this by hand: it was generated by git-svn init.
>
Try something like this:
[svn-remote "svn"]
# trunk
fetch = trunk:refs/remotes/trunk
# branches
fetch = branches/category-a/branch_a:refs/remotes/svn/branch_a
fetch = branches/category-a/branch_b:refs/remotes/svn/branch_b
fetch = branches/category-b/branch_c:refs/remotes/svn/branch_c
# tags
tags = tags/*:refs/remotes/tags/*
(Not sure if wildcards will work here, but I'm sure you could experiment and try
it out :-)
-Peter
^ permalink raw reply
* Re: [PATCH] Functions for updating refs.
From: Junio C Hamano @ 2007-09-04 17:52 UTC (permalink / raw)
To: Carlos Rica; +Cc: git, Johannes Schindelin
In-Reply-To: <46DD6020.4050401@gmail.com>
Carlos Rica <jasampler@gmail.com> writes:
> diff --git a/refs.c b/refs.c
> index 09a2c87..4fd5065 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1455,3 +1455,35 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)
> {
> return do_for_each_reflog("", fn, cb_data);
> }
> +
> +int update_ref_or_die(const char *action, const char *refname,
> + const unsigned char *sha1,
> + const unsigned char *oldval, int flags)
> +{
> + static struct ref_lock *lock;
> + lock = lock_any_ref_for_update(refname, oldval, flags);
> + if (!lock)
> + die("Cannot lock the ref '%s'.", refname);
> + if (write_ref_sha1(lock, sha1, action) < 0)
> + die("Cannot update the ref '%s'.", refname);
> + return 0;
> +}
> +
> +int update_ref_or_error(const char *action, const char *refname,
> + const unsigned char *sha1,
> + const unsigned char *oldval, int quiet)
> +{
> + static struct ref_lock *lock;
> + lock = lock_any_ref_for_update(refname, oldval, 0);
> + if (!lock) {
> + if (!quiet)
> + error("Cannot lock the ref '%s'.", refname);
> + return 1;
> + }
> + if (write_ref_sha1(lock, sha1, action) < 0) {
> + if (!quiet)
> + error("Cannot update the ref '%s'.", refname);
> + return 1;
> + }
> + return 0;
> +}
This makes me wonder three things:
- Why doesn't "or_error" side allow "flags" as "or_die" one?
Could the 'quiet' option become part of "flags" perhaps?
- They look quite similar. Is it a good idea to refactor them
further, or they are so small it does not matter?
- Why isn't lock released with unlock_ref()?
^ permalink raw reply
* Re: git-svn and a nested branches folder
From: Russ Brown @ 2007-09-04 18:03 UTC (permalink / raw)
To: git
In-Reply-To: <20070904174006.GB4538@xp.machine.xx>
Peter Baumann wrote:
> On Tue, Sep 04, 2007 at 10:21:22AM -0500, Russ Brown wrote:
>> David Kastrup wrote:
>>> Russ Brown <pickscrape@gmail.com> writes:
>>>
>>>> David Kastrup wrote:
>>>>> Russ Brown <pickscrape@gmail.com> writes:
>>>>>
>>>>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>>>>> think it's because the repository doesn't store branches as a flat list
>>>>>> directly under the 'branches' directory.
>>>>>>
>>>>>> Basically, we have a structure like this:
>>>>>>
>>>>>> |
>>>>>> +-trunk
>>>>>> +-tags
>>>>>> +-branches
>>>>>> + category-a
>>>>>> + branch-a
>>>>>> + branch-b
>>>>>> + category-b
>>>>>> + branch-c
>>>>>> + branch-d
>>>>>>
>>>>>> etc. category-a and category-b are simple directories created using svn
>>>>>> mkdir. The branches are created using svn cp.
>>>>>>
>>>>>> It helps us to organise the branches better, but the rationale is
>>>>>> besides the point. The problem is that git-svn seems to want to
>>>>>> treat category-a and category-b as branches, which isn't right at
>>>>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>>>>> that occur in these directories and creates a lot of entries in
>>>>>> unhandled.log.
>>>>> So what did you specify in your .git/config file regarding the svn
>>>>> structure?
>>>> I specified the 'branches' directory, but that's because earlier in
>>>> the life of the repo we did just do the flat branch layout, but
>>>> decided to make it more structured once that got unwieldy.
>>> Cough, cough. _What_ did you specify in your .git/config file
>>> regarding the svn structure? Please quote the section.
>>>
>> Erm, sorry.
>>
>> [svn-remote "svn"]
>> url = svn://svn.<name>.com
>> fetch = trunk:refs/remotes/trunk
>> branches = branches/*:refs/remotes/*
>> tags = tags/*:refs/remotes/tags/*
>>
>> (URL changed in case it annoys my employers)
>>
>> I didn't write this by hand: it was generated by git-svn init.
>>
>
> Try something like this:
>
> [svn-remote "svn"]
> # trunk
> fetch = trunk:refs/remotes/trunk
>
> # branches
> fetch = branches/category-a/branch_a:refs/remotes/svn/branch_a
> fetch = branches/category-a/branch_b:refs/remotes/svn/branch_b
> fetch = branches/category-b/branch_c:refs/remotes/svn/branch_c
>
> # tags
> tags = tags/*:refs/remotes/tags/*
>
>
> (Not sure if wildcards will work here, but I'm sure you could experiment and try
> it out :-)
>
Thanks a lot! I'll try it out.
I wonder if I'm going to have to fetch in stages here. i.e. configuring
the branches for the original layout, fetching up the point where the
layout changed a bit and reconfigure, fetch some more, tweak again etc.
> -Peter
--
Russ
^ permalink raw reply
* Re: Git's database structure
From: Mike Hommey @ 2007-09-04 18:04 UTC (permalink / raw)
To: Jon Smirl; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <9e4733910709041044r71264346n341d178565dd0521@mail.gmail.com>
On Tue, Sep 04, 2007 at 01:44:47PM -0400, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 9/4/07, Junio C Hamano <gitster@pobox.com> wrote:
> > "Jon Smirl" <jonsmirl@gmail.com> writes:
> >
> > > Another way of looking at the problem,
> > >
> > > Let's build a full-text index for git. You put a string into the index
> > > and it returns the SHAs of all the file nodes that contain the string.
> > > How do I recover the path names of these SHAs?
> >
> > That question does not make much sense without specifying "which
> > commit's path you are talking about".
> >
> > If you want to encode such "contextual information" in addition
> > to "contents", you could do so, but you essentially need to
> > record commit + pathname + mode bits + contents as "blob" and
> > hash that to come up with a name.
>
> I left the details out of the full-text example to make it more
> obvious that we can't recover the path names.
>
> Doing this type of analysis may point out that even more fields are
> missing from the blob table such as commit id.
>
> The current data store design is not very flexible. Databases solved
> the flexibility problem long ago. I'm just wondering if we should
> steal some good ideas out of the database world and apply them to git.
> Ten years from now we may have 100GB git databases and really wish we
> had more flexible ways of querying them.
>
> The reason databases don't encode the fields into the index is that
> you can only have a single index on the table if you do that.
> Databases do sometimes duplicate the field in both the index and the
> table. Databases also have the property that indexes are just a cache
> and can be dropped at any time.
The big difference between a database and git is that a database is a
general purpose tool. git has a much more restricted scope. As such, it
doesn't need *that much* flexibility.
Mike
^ permalink raw reply
* Re: Git's database structure
From: Junio C Hamano @ 2007-09-04 18:06 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910709041044r71264346n341d178565dd0521@mail.gmail.com>
"Jon Smirl" <jonsmirl@gmail.com> writes:
> On 9/4/07, Junio C Hamano <gitster@pobox.com> wrote:
>> "Jon Smirl" <jonsmirl@gmail.com> writes:
>>
>> > Another way of looking at the problem,
>> >
>> > Let's build a full-text index for git. You put a string into the index
>> > and it returns the SHAs of all the file nodes that contain the string.
>> > How do I recover the path names of these SHAs?
>>
>> That question does not make much sense without specifying "which
>> commit's path you are talking about".
>>
>> If you want to encode such "contextual information" in addition
>> to "contents", you could do so, but you essentially need to
>> record commit + pathname + mode bits + contents as "blob" and
>> hash that to come up with a name.
>
> I left the details out of the full-text example to make it more
> obvious that we can't recover the path names.
>
> Doing this type of analysis may point out that even more fields are
> missing from the blob table such as commit id.
Quite the contrary. You just illustrated why it is wrong to put
anything but contents in the blob.
The specialized indexing is a different issue. If you want to
have a full text index to answer "what paths in which commits
had this string?", then your database table would have columns
such as commit (sha-1), path (string) as values, indexed with
the search string.
Now the current set of "git" operation does not need to answer
that query, so we do not build nor maintain such an index that
nobody uses. But your application may benefit from such an
index, and as others said, nobody prevents you from building
one.
^ permalink raw reply
* Re: git clone over http
From: Robin Rosenberg @ 2007-09-04 18:19 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Josh England, git
In-Reply-To: <7vir6qnxuh.fsf@gitster.siamese.dyndns.org>
tisdag 04 september 2007 skrev Junio C Hamano:
> "Josh England" <jjengla@sandia.gov> writes:
>
> > On Sun, 2007-09-02 at 11:23 +0200, Robin Rosenberg wrote:
> >> git clone (1.5.3) with http is somewhat unreliable. I've noticed if
> >> can actually give
> >> me different versions of a branch each time I run it, eventually yielding
> >> the one I'm expecting. And now this:
> >
> > I don't see this behavior, but I do see other wierdness cloning via
> > http. Anytime I clone a repo (any repo) via http from behind the
> > firewall/proxy at work I never get any branches. 'git branch' only ever
> > shows 'master'. I can't checkout and work on any other branch. I can
> > work around the problem by cloning on a box that is more open, but was
> > wondering if anyone knows how to fix this.
>
> Well, what does "git branch -r" say?
>
> If you see the remote tracking branches out of your branch (aka
> "heads/") namespace, that has been the default for quite some
> time since 1.5.0 days.
>
> Perhaps "more open" one has ancient git that dumped remote
> tracking branches in your branch namespace, while your "behind
> firewall" one has not so ancient git?
The version where I noticed that strange behaviour was 1.5.3-rc6 on the client
and 1.5.1.4 on the server. The local repo was originally cloned with 1.5.something.
The interesting branch was a non-fastword one (i.e. "pu") and on the server I
was running git-update-server-info every 30 minutes. The pu branch on the server
was stable during the strange period. I'm not sure whether I got the odd behaviour
within those 30 minutes or not.
The url to the repo is http: //rosenberg.homelinux.net/repos/EGIT.git
I cannot http clone myself anymore it seems, but someone else may succeed better.
My curl is 7.16.0. Haven't tried any other version yet.
-- robin
^ permalink raw reply
* Re: Git's database structure
From: Andreas Ericsson @ 2007-09-04 18:51 UTC (permalink / raw)
To: Jon Smirl; +Cc: Julian Phillips, Git Mailing List
In-Reply-To: <9e4733910709041030ye912369nd574a5f78d3f521b@mail.gmail.com>
Jon Smirl wrote:
>
> I'm not proposing a change to data being stored in git, it is a
> proposal to consider the impacts of how this data has been normalized
> in the data store.
>
But to what end?
We all *know* the impacts:
* Excellent performance at what it does now.
* Currently zero capability to replace google as the #1 search engine.
Since replacing google's db was never, and will never, be the goal of
git, what is it you wish to achieve? Seriously, I'm dying to know, so
please tell me. If you have already and I'm too daft to understand it,
humor me and reiterate :-)
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: git pull fails with http urls?
From: Junio C Hamano @ 2007-09-04 19:17 UTC (permalink / raw)
To: Richard Purdie; +Cc: Git Mailing List, Petr Baudis
In-Reply-To: <1188915053.5705.12.camel@localhost.localdomain>
Pasky seems to have a locally modified version at repo.or.cz to
add HEAD to the result of update-server-info, but as far as I
can see, update-server-info never placed HEAD in the result, and
I do not think ls-remote compensated it by adding HEAD by hand,
so that change is fixing the symptom rather than fixing the
regression, but I am not sure where the true regression is.
Most likely we broke git-fetch while trying to deal with
repositories with insanely large number of refs (aka
"fetch--tool partial rewrite in C").
Having said all that, I think it is a good idea nevertheless to
add HEAD information across all transports, like Pasky's patch
does. Currently http/ftp/rsync do not report HEAD while git
native protocols do.
Pasky, care to submit "server-info: Add HEAD to info/refs" for
inclusion?
^ permalink raw reply
* Re: [PATCH] Add post-merge hook.
From: Josh England @ 2007-09-04 19:36 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vmyw2ny05.fsf@gitster.siamese.dyndns.org>
On Tue, 2007-09-04 at 10:25 -0700, Junio C Hamano wrote:
> "Josh England" <jjengla@sandia.gov> writes:
>
> >> Two questions.
> >>
> >> * Do you want to run the post-merge hook even for a squash
> >> merge?
> >
> > Yes. I'd like to run it at any time that the working tree might be
> > updated.
>
> If that is the case, perhaps your hook may want to get a
> parameter to tell it what kind of "git-merge" invocation it was?
> Squash merge does not even advance the HEAD and is of a very
> different nature from a normal merge.
OK. Should it just pass in a flag (squash or normal), or are there
other merge types it should need to know about.
-JE
^ permalink raw reply
* Re: Git's database structure
From: Reece Dunn @ 2007-09-04 19:44 UTC (permalink / raw)
To: Mike Hommey, Jon Smirl, Junio C Hamano, Git Mailing List
In-Reply-To: <20070904180429.GA626@glandium.org>
On 04/09/07, Mike Hommey <mh@glandium.org> wrote:
> On Tue, Sep 04, 2007 at 01:44:47PM -0400, Jon Smirl <jonsmirl@gmail.com> wrote:
> > The reason databases don't encode the fields into the index is that
> > you can only have a single index on the table if you do that.
> > Databases do sometimes duplicate the field in both the index and the
> > table. Databases also have the property that indexes are just a cache
> > and can be dropped at any time.
>
> The big difference between a database and git is that a database is a
> general purpose tool. git has a much more restricted scope. As such, it
> doesn't need *that much* flexibility.
Databases are designed to be efficient at storing and accessing large
amounts of data. The key thing about a database is that it does not
track the *history* of the data it is storing. This is the main
problem with using a database as a metadata storage facility.
Modern source control systems such as Perforce (and possibly
Subversion), use a database to track metadata such as branch/merge
history, user data and so on. This, IMHO is a huge weakness of these
SCM systems. It is impossible to fully roll back to a given point in
time, because that metadata is stored independently of the file
content tracking.
Git *is not a database*. This is fundamental to understanding how git
works. Git stores *all* of its data in a Directed Acyclic Graph (with
the exception of the pointers to tag and the current head of each
branch, that it stores locally in the .git directory). Read
http://eagain.net/articles/git-for-computer-scientists/ for more
information on this.
What this means is that for any commit, git has all the information it
needs about the repository at that point in time. It doesn't need
anything else. If you then store information in a database, you lose
having the complete picture at any point in the history of the
repository.
- Reece
^ permalink raw reply
* Re: How-to setup an empty remote repository?
From: Jan Hudec @ 2007-09-04 19:47 UTC (permalink / raw)
To: Wink Saville; +Cc: git
In-Reply-To: <d4cf37a60708251400t17b0a097t91f77cbb4e996810@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]
On Sat, Aug 25, 2007 at 14:00:33 -0700, Wink Saville wrote:
> Hello,
>
> I thought I'd try to setup a shared "empty" remote repository and then
> clone it and then push the initial commit from another maching.
> This failed as shown below:
>
> $ cd ~/git-repos
> $ mkdir test.git
> $ cd test.git
> $ git --bare init --shared
> Initialized empty shared Git repository in /home/wink/git-repos/test.git/
> $ git-daemon --reuseaddr --verbose --base-path=/home/wink/git-repos \
> --export-all --enable=receive-pack
>
> On my other machine:
>
> $ git clone git://192.168.0.8/test.git
> Initialized empty Git repository in /home/wink/prgs/test/.git/
> fatal: no matching remote head
> fetch-pack from 'git://192.168.0.8/test.git' failed.
> $
Did it configure the repository though? Like setting the url and tracking
branches for origin? If it did, it actually did all you needed, so you can
count it as success (and the message should talk about having nothing to do
rather than failing).
--
Jan 'Bulb' Hudec <bulb@ucw.cz>
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: [PATCH] Add post-merge hook.
From: Junio C Hamano @ 2007-09-04 20:03 UTC (permalink / raw)
To: Josh England; +Cc: Junio C Hamano, git
In-Reply-To: <1188934573.6192.35.camel@beauty>
"Josh England" <jjengla@sandia.gov> writes:
> On Tue, 2007-09-04 at 10:25 -0700, Junio C Hamano wrote:
>> "Josh England" <jjengla@sandia.gov> writes:
>>
>> >> Two questions.
>> >>
>> >> * Do you want to run the post-merge hook even for a squash
>> >> merge?
>> >
>> > Yes. I'd like to run it at any time that the working tree might be
>> > updated.
>>
>> If that is the case, perhaps your hook may want to get a
>> parameter to tell it what kind of "git-merge" invocation it was?
>> Squash merge does not even advance the HEAD and is of a very
>> different nature from a normal merge.
>
> OK. Should it just pass in a flag (squash or normal), or are there
> other merge types it should need to know about.
I suspect you have thought abuot the issues involved longer than
I have ;-), so you should take whatever I say with grain of
salt, but I think you would also want to know fast-forwards and
up-to-dates if the hook wants to be generic, not "for Josh's
workflow only".
^ permalink raw reply
* Re: Git's database structure
From: David Tweed @ 2007-09-04 20:17 UTC (permalink / raw)
To: Jon Smirl; +Cc: Andreas Ericsson, Git Mailing List
In-Reply-To: <9e4733910709040919u3d252b91s2785ed4d20086c88@mail.gmail.com>
On 9/4/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> without building another mapping table or a brute force search. I keep
> using Google as an example, Google is indexing hierarchical URLs but
> they do not use a hierarchical index to do it.
It might help the discussion if you could point to a reference,
preferably one that discusses the trade-offs in the design, with more
concrete details about what google or other search engines actually
do. It would be particularly useful if it addressed issues of
1. the type of queries the representation is optimised for.
2. consistency requirements. (Can a search engine use different data
structures if they improve average performance at the cost of
occasional inconsistency/lossage?)
Finally, this design space is not totally unexplored, for example,
http://plan9.bell-labs.com/sys/doc/venti/venti.html
AFAICS they only use SHA-1 for blocks within files (although this
might be misreading the paper) so presumably they'd have knowledge
about the trade-offs.
--
cheers, dave tweed__________________________
david.tweed@gmail.com
Rm 124, School of Systems Engineering, University of Reading.
"we had no idea that when we added templates we were adding a Turing-
complete compile-time language." -- C++ standardisation committee
^ permalink raw reply
* Re: .gitignore, .gitattributes, .gitmodules, .gitprecious?,.gitacls? etc.
From: Jan Hudec @ 2007-09-04 20:23 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Dmitry Kakurin, Petr Baudis, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0708280945350.28586@racer.site>
[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]
On Tue, Aug 28, 2007 at 09:49:47 +0100, Johannes Schindelin wrote:
> Hi,
>
> On Mon, 27 Aug 2007, Dmitry Kakurin wrote:
>
> > Here is the problem: we need to apply crlf attributes to a file. We
> > could have .gitattributes both in the index and in the worktree.
> > Which one do we use?
> > In general .gitattributes file could be (U)nchanged, (C)hanged, (NP)
> > NotPresent in each place.
>
> I do not see these cases. You can have these cases, basically:
>
> - .gitattributes in worktree (then it does not matter what else we have),
> - .gitattributes not in the worktree, but in the index (then that is taken)
>
> In the latter case, there could be conflicts _in_ .gitattributes, in which
> case those .gitattributes are ignored.
>
> I do not see any problem with that.
I do.
IMNSHO it should be the other way around:
.gitattributes in index, than index version is used.
.gitattributes not in index, but in worktree, than that tree version is used.
Why? Because when you check out another version, the .gitattributes commited
in that version need to be applied, since it might be different from whatever
is currently in the tree.
In the other direction, the tree version seems to make more sense, but in
reality it does not. If you do a partial commit of a single file, than the
index version gets into the commit, so that should better be the version used
to store the file. On the other hand if you change .gitattributes, the normal
commit rules are that you need to add it for commit, so needing to add it to
make it effective goes well together with it. update-index would need to
always handle .git* before other entries to make add . and commit -a work
correctly.
The case for worktree only is for cases when you for some reason want to have
local .gitattributes. Though I am not sure that should actually work, because
you couldn't have local .gitattributes if there is versioned version.
.git/info/attributes would be better for that. Which gets us back to "always
use .gitattributes from *index*" (and read it from tree before other files
when adding it).
--
Jan 'Bulb' Hudec <bulb@ucw.cz>
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox