git fetch: where are the downloaded objects stored?

git fetch: where are the downloaded objects stored?

[Paolo Ciarrocchi]

Hi all,
I'm thinking about writing some documentation including diagrams about
the common git operation like pull, fetch, merge and so on in order
to give a "visual" learning procedure about:
- local and remote branches
- index
- checkout dir
-

Re: git fetch: where are the downloaded objects stored?

[Matthieu Moy]

"Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:

> What is still not clear to me is where git is storing
> the objects downloaded during a git fetch operation.

git fetch downloads the objects to the object database, i.e.
.git/objects/, usualy as a pack.

Now, just "having" the objects locally doesn't change much, an object
starts being really interesting if you have a reference (typically, a
branch) pointing to it. "git fetch" will update the remote references
(for example, origin/master), but won't touch the working tree, and
won't do any merge, fast-forward or not, to a local branch. So, most
likely, "git fetch" will be followed by either a merge or a rebase.

-- 
Matthieu

Re: git fetch: where are the downloaded objects stored?

[Paolo Ciarrocchi]

On Mon, Mar 3, 2008 at 2:14 PM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
>
>  > What is still not clear to me is where git is storing
>  > the objects downloaded during a git fetch operation.
>
>  git fetch downloads the objects to the object database, i.e.
>  .git/objects/, usualy as a pack.

Ok, thanks.

>  Now, just "having" the objects locally doesn't change much, an object
>  starts being really interesting if you have a reference (typically, a
>  branch) pointing to it. "git fetch" will update the remote references
>  (for example, origin/master), but won't touch the working tree, and
>  won't do any merge, fast-forward or not, to a local branch. So, most
>  likely, "git fetch" will be followed by either a merge or a rebase.

Yes, your explanation perfectly match my understanding.

I was, and I'm still, confused by the following sentence in the git merge
documentation:
"A merge is always between the current HEAD and one or more remote
branch heads"

If I think to the following scenario:

 git fetch
 git diff (to look at what I downloaded via git fetch)
 gir merge

I don't see a merge between the current HEAD and a _remote_ branch.

Ciao,
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/

Re: git fetch: where are the downloaded objects stored?

[Matthieu Moy]

"Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:

> "A merge is always between the current HEAD and one or more remote
> branch heads"

I think this is just wrong. Would this be correct?

diff --git a/Documentation/git-merge.txt b/Documentation/git-merge.txt
index 0c9ad7f..e46dea1 100644
--- a/Documentation/git-merge.txt
+++ b/Documentation/git-merge.txt
@@ -68,7 +68,7 @@ HOW MERGE WORKS
 ---------------
 
 A merge is always between the current `HEAD` and one or more
-remote branch heads, and the index file must exactly match the
+branch heads (remote or local), and the index file must exactly match the
 tree of `HEAD` commit (i.e. the contents of the last commit) when
 it happens.  In other words, `git-diff --cached HEAD` must
 report no changes.


-- 
Matthieu

Re: git fetch: where are the downloaded objects stored?

[Paolo Ciarrocchi]

On Mon, Mar 3, 2008 at 4:21 PM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
>
> > "A merge is always between the current HEAD and one or more remote
> > branch heads"
>
> I think this is just wrong. Would this be correct?

Sounds better than the original document,
however I'm still having some problems in visualizing what happens
when I type "git fetch" followed by "git merge".

> diff --git a/Documentation/git-merge.txt b/Documentation/git-merge.txt
> index 0c9ad7f..e46dea1 100644
> --- a/Documentation/git-merge.txt
> +++ b/Documentation/git-merge.txt
> @@ -68,7 +68,7 @@ HOW MERGE WORKS
>  ---------------
>
>  A merge is always between the current `HEAD` and one or more
> -remote branch heads, and the index file must exactly match the
> +branch heads (remote or local), and the index file must exactly match the

When I run the command git fetch the objects are downloaded from the remote
branch and locally stored in the objects database.
Both the working tree and index are not touched by this operation.
Is this correct?

How can I look to what I just downloaded?
Should I simply do a git diff?

Backing to the documentation, your proposal is:
 A merge is always between the current `HEAD` and one or more
 branch heads (remote or local), and the index file must exactly match the

In case of a git fetch + git merge the merge is between the current
`HEAD` and the
downloaded objects. Is correct to define it `branch heads`?

Maybe (read it: for sure) I'm a bit confused by the git terminology
but I really feel that
other newbies are not easily understanding this process.

Thanks.

ciao,
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/

Re: git fetch: where are the downloaded objects stored?

[Nicolas Pitre]

On Mon, 3 Mar 2008, Paolo Ciarrocchi wrote:

> On Mon, Mar 3, 2008 at 4:21 PM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> > "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
> >
> > > "A merge is always between the current HEAD and one or more remote
> > > branch heads"
> >
> > I think this is just wrong. Would this be correct?
> 
> Sounds better than the original document,
> however I'm still having some problems in visualizing what happens
> when I type "git fetch" followed by "git merge".
> 
> > diff --git a/Documentation/git-merge.txt b/Documentation/git-merge.txt
> > index 0c9ad7f..e46dea1 100644
> > --- a/Documentation/git-merge.txt
> > +++ b/Documentation/git-merge.txt
> > @@ -68,7 +68,7 @@ HOW MERGE WORKS
> >  ---------------
> >
> >  A merge is always between the current `HEAD` and one or more
> > -remote branch heads, and the index file must exactly match the
> > +branch heads (remote or local), and the index file must exactly match the

Yes, this is much better.  A merge may occur with any other branches, 
local or remote, or even with a tag, or any other random commit 
reference for that matter.

> When I run the command git fetch the objects are downloaded from the remote
> branch and locally stored in the objects database.
> Both the working tree and index are not touched by this operation.
> Is this correct?

Yes.  The fetch operation will figure out, with the remote machine, what 
is the set of objects that you already have and the set that you don't 
have so the remote machine will create and send you a pack of only the 
objects you're missing.  And the remote machine will also reduce it to 
deltas against objects that you already have when possible so the 
transferred pack is even smaller.  Once that pack has successfully been 
received, then the branch head for which this pack was requested will be 
updated to point at the latest commit for that branch.

> How can I look to what I just downloaded?
> Should I simply do a git diff?

If you have reflog enabled (it should be by default) then a good thing 
to remember is the @{1} notation.  For example, if the fetch updated the 
origin/master branch, then origin/master@{1} is what your origin/master 
was before being updated.  To see the difference between the previous 
and the current state of origin/master, you can do:

	git diff origin/master@{1}..origin/master

Or to see the list of new commits:

	git log origin/master@{1}..origin/master

	git log -p origin/master@{1}..origin/master

Etc.

This notation is a bit obnoxious and the re were suggestions about 
addind the equivalent origin/master@{1..} but that didn't materialize 
yet.

> Backing to the documentation, your proposal is:
>  A merge is always between the current `HEAD` and one or more
>  branch heads (remote or local), and the index file must exactly match the
> 
> In case of a git fetch + git merge the merge is between the current
> `HEAD` and the
> downloaded objects. Is correct to define it `branch heads`?

A merge doesn't happen between a branch and some objects.  Please don't 
see it that way.  Objects are at a lower level of abstraction.  What a 
fetch does is to make sure your version of a branch (say origin/master) 
matches the remote version of the branch "master" on server "origin".  
If you happen to already have all the needed objects already, then no 
objects will be transferred and only the branch reference will be 
updated.

The merge operation really works at the commit graph level in order to 
jointwo or more branches together. Objects associated to the involved 
branches are only checked so to make sure the merging of the specified 
branches does not create a conflict (and to fix it otherwise).  If a 
merge conflict is fixed (either manually or automatically) then new 
objects corresponding to the modified files are locally created but the 
previously existing objects remain unchanged.  But object handling 
during a merge is really a low level thing.

> Maybe (read it: for sure) I'm a bit confused by the git terminology
> but I really feel that
> other newbies are not easily understanding this process.

I suggest you have a look at the following article:

	http://eagain.net/articles/git-for-computer-scientists/ 

It is really well written, with the right level of vulgarization to make 
the Git concept really obvious very quickly.


Nicolas

Re: git fetch: where are the downloaded objects stored?

[Paolo Ciarrocchi]

On Mon, Mar 3, 2008 at 5:29 PM, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 3 Mar 2008, Paolo Ciarrocchi wrote:
>
> > On Mon, Mar 3, 2008 at 4:21 PM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> > > "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
> > >
> > > > "A merge is always between the current HEAD and one or more remote
> > > > branch heads"
> > >
> > > I think this is just wrong. Would this be correct?
> >
> > Sounds better than the original document,
> > however I'm still having some problems in visualizing what happens
> > when I type "git fetch" followed by "git merge".
> >
> > > diff --git a/Documentation/git-merge.txt b/Documentation/git-merge.txt
> > > index 0c9ad7f..e46dea1 100644
> > > --- a/Documentation/git-merge.txt
> > > +++ b/Documentation/git-merge.txt
> > > @@ -68,7 +68,7 @@ HOW MERGE WORKS
> > >  ---------------
> > >
> > >  A merge is always between the current `HEAD` and one or more
> > > -remote branch heads, and the index file must exactly match the
> > > +branch heads (remote or local), and the index file must exactly match the
>
> Yes, this is much better.  A merge may occur with any other branches,
> local or remote, or even with a tag, or any other random commit
> reference for that matter.

It's probably a good idea to mention the tag as well.
Something like:
  A merge is always between the current `HEAD` and one or more
  branch heads (remote or local) or tags, and the index file must
exactly match the

> > When I run the command git fetch the objects are downloaded from the remote
> > branch and locally stored in the objects database.
> > Both the working tree and index are not touched by this operation.
> > Is this correct?
>
> Yes.  The fetch operation will figure out, with the remote machine, what
> is the set of objects that you already have and the set that you don't
> have so the remote machine will create and send you a pack of only the
> objects you're missing.  And the remote machine will also reduce it to
> deltas against objects that you already have when possible so the
> transferred pack is even smaller.  Once that pack has successfully been
> received, then the branch head for which this pack was requested will be
> updated to point at the latest commit for that branch.

OK, that's now completely clear and understood.

> > How can I look to what I just downloaded?
> > Should I simply do a git diff?
>
> If you have reflog enabled (it should be by default) then a good thing
> to remember is the @{1} notation.  For example, if the fetch updated the
> origin/master branch, then origin/master@{1} is what your origin/master
> was before being updated.  To see the difference between the previous
> and the current state of origin/master, you can do:
>
>        git diff origin/master@{1}..origin/master
>
> Or to see the list of new commits:
>
>        git log origin/master@{1}..origin/master
>
>        git log -p origin/master@{1}..origin/master
>
> Etc.

Very nice, I didn't find in the documentation.
I'll read again the documents and if needed, I'll propose some new text.

> This notation is a bit obnoxious and the re were suggestions about
> addind the equivalent origin/master@{1..} but that didn't materialize
> yet.

Mybe it's just me but wouldn't be very nice to have a simple command
to look at what data have been used for updating the currente branch?
i.e.
git fetch
git diff -- fetch (which is an alias of git diff
origin/master@{1}..origin/master)

And how about a repository which have reflogs disabled?

> > Backing to the documentation, your proposal is:
> >  A merge is always between the current `HEAD` and one or more
> >  branch heads (remote or local), and the index file must exactly match the
> >
> > In case of a git fetch + git merge the merge is between the current
> > `HEAD` and the
> > downloaded objects. Is correct to define it `branch heads`?
>
> A merge doesn't happen between a branch and some objects.  Please don't
> see it that way.  Objects are at a lower level of abstraction.  What a
> fetch does is to make sure your version of a branch (say origin/master)
> matches the remote version of the branch "master" on server "origin".
> If you happen to already have all the needed objects already, then no
> objects will be transferred and only the branch reference will be
> updated.

OK.

> The merge operation really works at the commit graph level in order to
> jointwo or more branches together. Objects associated to the involved
> branches are only checked so to make sure the merging of the specified
> branches does not create a conflict (and to fix it otherwise).  If a
> merge conflict is fixed (either manually or automatically) then new
> objects corresponding to the modified files are locally created but the
> previously existing objects remain unchanged.  But object handling
> during a merge is really a low level thing.
>
> > Maybe (read it: for sure) I'm a bit confused by the git terminology
> > but I really feel that
> > other newbies are not easily understanding this process.
>
> I suggest you have a look at the following article:
>
>        http://eagain.net/articles/git-for-computer-scientists/
>
> It is really well written, with the right level of vulgarization to make
> the Git concept really obvious very quickly.

I read it, very intersting.

Thank you Nicolas.

Ciao,
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/

Re: git fetch: where are the downloaded objects stored?

[Sergei Organov]

"Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
> On Mon, Mar 3, 2008 at 5:29 PM, Nicolas Pitre <nico@cam.org> wrote:
>> On Mon, 3 Mar 2008, Paolo Ciarrocchi wrote:
>>
>> > On Mon, Mar 3, 2008 at 4:21 PM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:

[...]

>> > How can I look to what I just downloaded?
>> > Should I simply do a git diff?
>>
>> If you have reflog enabled (it should be by default) then a good thing
>> to remember is the @{1} notation.  For example, if the fetch updated the
>> origin/master branch, then origin/master@{1} is what your origin/master
>> was before being updated.  To see the difference between the previous
>> and the current state of origin/master, you can do:
>>
>>        git diff origin/master@{1}..origin/master
>>
>> Or to see the list of new commits:
>>
>>        git log origin/master@{1}..origin/master
>>
>>        git log -p origin/master@{1}..origin/master
>>
>> Etc.
>
> Very nice, I didn't find in the documentation.
> I'll read again the documents and if needed, I'll propose some new text.
>
>> This notation is a bit obnoxious and the re were suggestions about
>> addind the equivalent origin/master@{1..} but that didn't materialize
>> yet.
>
> Mybe it's just me but wouldn't be very nice to have a simple command
> to look at what data have been used for updating the currente branch?
> i.e.
> git fetch
> git diff -- fetch (which is an alias of git diff
> origin/master@{1}..origin/master)
>
> And how about a repository which have reflogs disabled?

I'm also a newbie, and I think you are on a wrong road. Usually it's not
that interesting what was downloaded. What is interesting is what is on
'origin/master' that is still missing from 'master'. For this I think
you have:

$ git log master..origin

-- Sergei.

Re: git fetch: where are the downloaded objects stored?

[Nicolas Pitre]

On Mon, 3 Mar 2008, Sergei Organov wrote:
> "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
> > On Mon, Mar 3, 2008 at 5:29 PM, Nicolas Pitre <nico@cam.org> wrote:
> >> If you have reflog enabled (it should be by default) then a good thing
> >> to remember is the @{1} notation.  For example, if the fetch updated the
> >> origin/master branch, then origin/master@{1} is what your origin/master
> >> was before being updated.  To see the difference between the previous
> >> and the current state of origin/master, you can do:
> >>
> >>        git diff origin/master@{1}..origin/master
> >>
> >> Or to see the list of new commits:
> >>
> >>        git log origin/master@{1}..origin/master
> >>
> >>        git log -p origin/master@{1}..origin/master
> >>
> >> Etc.
> >
> > Very nice, I didn't find in the documentation.
> > I'll read again the documents and if needed, I'll propose some new text.
> >
> >> This notation is a bit obnoxious and the re were suggestions about
> >> addind the equivalent origin/master@{1..} but that didn't materialize
> >> yet.
> >
> > Mybe it's just me but wouldn't be very nice to have a simple command
> > to look at what data have been used for updating the currente branch?
> > i.e.
> > git fetch
> > git diff -- fetch (which is an alias of git diff
> > origin/master@{1}..origin/master)
> >
> > And how about a repository which have reflogs disabled?
> 
> I'm also a newbie, and I think you are on a wrong road. Usually it's not
> that interesting what was downloaded. What is interesting is what is on
> 'origin/master' that is still missing from 'master'. For this I think
> you have:
> 
> $ git log master..origin

Yes, this is indeed what most people should be doing.  But if you're 
tracking some remote repository with no intention of merging them in 
your master branch by default, then it is often a nice thing to be able 
to see what has changed in those tracked branches since the last time 
you fetched them.  This is where the reflog is really handy as it 
records all fetch points you did.


Nicolas

Re: git fetch: where are the downloaded objects stored?

[Nicolas Pitre]

On Mon, 3 Mar 2008, Paolo Ciarrocchi wrote:

> On Mon, Mar 3, 2008 at 5:29 PM, Nicolas Pitre <nico@cam.org> wrote:
> > If you have reflog enabled (it should be by default) then a good thing
> > to remember is the @{1} notation.  For example, if the fetch updated the
> > origin/master branch, then origin/master@{1} is what your origin/master
> > was before being updated.  To see the difference between the previous
> > and the current state of origin/master, you can do:
> >
> >        git diff origin/master@{1}..origin/master
> >
> > Or to see the list of new commits:
> >
> >        git log origin/master@{1}..origin/master
> >
> >        git log -p origin/master@{1}..origin/master
> >
> > Etc.
> 
> Very nice, I didn't find in the documentation.
> I'll read again the documents and if needed, I'll propose some new text.

I found the reflog notation burried in git-rev-parse.txt.  Maybe some 
example usage could be added to more frequently used commands.

> > This notation is a bit obnoxious and the re were suggestions about
> > addind the equivalent origin/master@{1..} but that didn't materialize
> > yet.
> 
> Mybe it's just me but wouldn't be very nice to have a simple command
> to look at what data have been used for updating the currente branch?
> i.e.
> git fetch
> git diff -- fetch (which is an alias of git diff
> origin/master@{1}..origin/master)

You still think in terms of "data used to update a branch".  There is no 
such direct relation between the fetched data (objects) and the updated 
branch heads.  What you do when listing the commits found between 
origin/master@{1} and origin/master@{0} (or simply origin/master wich 
is equivalent to origin/master@{0}) does not necessarily correspond to 
the data you received during the fetch.  For example, it is possible 
that half of those commits were already part of another branch in your 
repository, hence the fetch operation won't download them again.  Still, 
the origin/master branch now has acquired them all which wasn't the case 
before.  To really _see_ the actually received data during a fetch 
requires internal Git knowledge and digging in the pack directory -- 
there is no (need for any) UI for that.

> And how about a repository which have reflogs disabled?

Well, just re-enable them.  ;-)

Since they're on by default, if you turned them off then you surely know 
what you're doing and why.


Nicolas

Re: git fetch: where are the downloaded objects stored?

[Paolo Ciarrocchi]

On Mon, Mar 3, 2008 at 7:07 PM, Nicolas Pitre <nico@cam.org> wrote:
[...]
>  You still think in terms of "data used to update a branch".  There is no
>  such direct relation between the fetched data (objects) and the updated
>  branch heads.  What you do when listing the commits found between
>  origin/master@{1} and origin/master@{0} (or simply origin/master wich
>  is equivalent to origin/master@{0}) does not necessarily correspond to
>  the data you received during the fetch.  For example, it is possible
>  that half of those commits were already part of another branch in your
>  repository, hence the fetch operation won't download them again.  Still,
>  the origin/master branch now has acquired them all which wasn't the case
>  before.  To really _see_ the actually received data during a fetch
>  requires internal Git knowledge and digging in the pack directory --
>  there is no (need for any) UI for that.

Yes, this time it was me using a completely wrong terminology.
What I was looking for is what Sergei suggested.

>  > And how about a repository which have reflogs disabled?
>
>  Well, just re-enable them.  ;-)
>
>  Since they're on by default, if you turned them off then you surely know
>  what you're doing and why.

Oh well, very true!

Maybe it's again just me which is still not used to git fetch (or my
mental translation of °fetch° in Italian)
but I found easier to "visualize" the commang "git remote update"
which in my setup alway means
"Fetch updates all the remotes in the repository"

Again, thank you for your comments and for your time.

Ciao,
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/

Re: git fetch: where are the downloaded objects stored?

[Jakub Narebski]

"Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:

> On Mon, Mar 3, 2008 at 2:14 PM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
>> "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> writes:
>>
>>> What is still not clear to me is where git is storing
>>> the objects downloaded during a git fetch operation.
>>
>>  git fetch downloads the objects to the object database, i.e.
>>  .git/objects/, usualy as a pack.

Downloading as individual objects ("exploded") or as a pack depends on
fetch.unpackLimit configuration.
 
>>  Now, just "having" the objects locally doesn't change much, an object
>>  starts being really interesting if you have a reference (typically, a
>>  branch) pointing to it. "git fetch" will update the remote references
>>  (for example, origin/master), but won't touch the working tree, and
>>  won't do any merge, fast-forward or not, to a local branch. So, most
>>  likely, "git fetch" will be followed by either a merge or a rebase.

What is important git-fetch _always_ stores what it fetched in
$GIT_DIR/FETCH_HEAD.

Then depending on configuration and command line options it updates
appropriate remote-tracking branches.  For example if you do one-shot
"git pull <remote> <branch>" git would not store fetched refs (heads)
in tracking branches: see git-fetch(1).
 
> Yes, your explanation perfectly match my understanding.
> 
> I was, and I'm still, confused by the following sentence in the git merge
> documentation:
>
>   "A merge is always between the current HEAD and one or more
>    remote branch heads"

This "remote" is here invalid. Merge can merge in any ref, not only
remote-tracking branches.  "git merge <branch>" would merge given
branch into current (HEAD) branch:

Before:

  1---2---3---4---5---6---7   <-- master <-- HEAD
               \
                \-A---B       <-- devel (or origin/master)

After "git merge devel"

  1---2---3---4---5---6---7---M   <-- master <-- HEAD
               \             /
                \-A---B-----/     <-- devel (or origin/master)

where 'M' is merge (multi-parent) commit.

> If I think to the following scenario:
> 
>  git fetch
>  git diff (to look at what I downloaded via git fetch)

Because you download _series_ of commits "git log" would be usually
more useful than "git diff".

You can use <branch>@{1} for position of branch before fetch, or even
ORIG_HEAD for position of HEAD branch before fetch (or pull).

>  git merge

I'm not sure if "git merge" without branch specified would work.
"git pull" should, so perhaps "git merge" would too.

"git pull" would merge 'first' branch, or branch specified in
branch/remote configuration.

-- 
Jakub Narebski
Poland
ShadeHawk on #git