[GSoC/RFC] Ideas on git fetch --deepen

[GSoC/RFC] Ideas on git fetch --deepen

[Dongcan Jiang]

Hi all,

My name is Dongcan Jiang. I am studying for my Master Degree at Peking
University
majoring in Computer Science. I have been using Git to manage my projects
for about half a year. It's really exciting that Git has been helping me make
revision control much more convenient. Therefore, I am very interested in
doing some works for Git in GSoC 2015.

I have submitted my v2 patch on a microproject recently, and I am waiting
for comments on v2 now.

In the meantime, I have been scanning ideas on the git gsoc page and
their related mails. I find that I am interested in most of them, especially
"git fetch --deepen" idea.

Here is my understanding about this idea.

Although "deepen" and "depth" have different behavior, "deepen" can be
achieved by "depth" with some extra calculation. If we know the
distance between "my history bottom" and "your tips", we can set the sum
of "deepen" step and this distance as "depth" step. Then we can reuse
the logics of processing "depth" to complete it.

Take graph in [1] as an example.

>  (upstream)
>   ---o---o---o---A---B
>
>  (you)
>                  A---B

the distance of "my history bottom"(A) and "your tips"(B) is 1, then
"git fetch --deepen=3" can be achieved by "git fetch --depth=4"

However, I am a little worried about whether this idea is enough for a
GSoC project. I hope you can give me some comments and suggestions.

[1] http://article.gmane.org/gmane.comp.version-control.git/212950

Thanks,
Dongcan

-- 
江东灿（Dongcan Jiang）
Team of Search Engine & Web Mining
School of Electronic Engineering & Computer Science
Peking University, Beijing, 100871, P.R.China

Re: [GSoC/RFC] Ideas on git fetch --deepen

[Dongcan Jiang]

Hi, all

After digging into how "git fetch" works, I have found that my previous
understanding was too rash. I'm sorry for that.

I find that the current workflow of "git fetch --depth" is as follows:

  1. 'fetch module' calls 'git-upload-pack service' via 'transport module'
     to fetch ref with `depth`.
  2. Such call is finished by pipe I/O.
  3. git-upload-pack receives arguments such as 'depth' to fetch commits,
     and send back to the caller.

Therefore, if we want to implement "--deepen" for "git fetch", we have to
modify the arguments protocol of git-upload-pack service by adding the
'shallow commit' hash.
Then, we can fetch `depth` commits before the 'shallow commit' in
git-upload-pack service.
Apparently, we have to output error message when the 'shallow commit' hash
is not in the repository.

However, I still have a question for [1]. I am not quite following this:

> If you want the new history leading to the updated tip,
> you can just say:
>
>    git fetch
>
> without any --depth nor --deepen option to end up with:
>
>  (you)
>                  A---B---C---D---E---F---...---W---X---Y---Z

As far as I know, git fetch --depth would fetch new history to the
local remotes' refs.
Does it mean that we have to change the behavior of "git fetch --depth"?

I hope you can give me some comments or suggestions, letting me know
whether I am in the right way.

Thanks.

[1] http://article.gmane.org/gmane.comp.version-control.git/212950

2015-03-07 14:32 GMT+08:00 Dongcan Jiang <dongcan.jiang@gmail.com>:
> Hi all,
>
> My name is Dongcan Jiang. I am studying for my Master Degree at Peking
> University
> majoring in Computer Science. I have been using Git to manage my projects
> for about half a year. It's really exciting that Git has been helping me make
> revision control much more convenient. Therefore, I am very interested in
> doing some works for Git in GSoC 2015.
>
> I have submitted my v2 patch on a microproject recently, and I am waiting
> for comments on v2 now.
>
> In the meantime, I have been scanning ideas on the git gsoc page and
> their related mails. I find that I am interested in most of them, especially
> "git fetch --deepen" idea.
>
> Here is my understanding about this idea.
>
> Although "deepen" and "depth" have different behavior, "deepen" can be
> achieved by "depth" with some extra calculation. If we know the
> distance between "my history bottom" and "your tips", we can set the sum
> of "deepen" step and this distance as "depth" step. Then we can reuse
> the logics of processing "depth" to complete it.
>
> Take graph in [1] as an example.
>
>>  (upstream)
>>   ---o---o---o---A---B
>>
>>  (you)
>>                  A---B
>
> the distance of "my history bottom"(A) and "your tips"(B) is 1, then
> "git fetch --deepen=3" can be achieved by "git fetch --depth=4"
>
> However, I am a little worried about whether this idea is enough for a
> GSoC project. I hope you can give me some comments and suggestions.
>
> [1] http://article.gmane.org/gmane.comp.version-control.git/212950
>
> Thanks,
> Dongcan
>
> --
> 江东灿（Dongcan Jiang）
> Team of Search Engine & Web Mining
> School of Electronic Engineering & Computer Science
> Peking University, Beijing, 100871, P.R.China



-- 
江东灿（Dongcan Jiang）
Team of Search Engine & Web Mining
School of Electronic Engineering & Computer Science
Peking University, Beijing, 100871, P.R.China

Re: [GSoC/RFC] Ideas on git fetch --deepen

[Duy Nguyen]

On Sun, Mar 8, 2015 at 3:57 PM, Dongcan Jiang <dongcan.jiang@gmail.com> wrote:
> Hi, all
>
> After digging into how "git fetch" works, I have found that my previous
> understanding was too rash. I'm sorry for that.
>
> I find that the current workflow of "git fetch --depth" is as follows:
>
>   1. 'fetch module' calls 'git-upload-pack service' via 'transport module'
>      to fetch ref with `depth`.
>   2. Such call is finished by pipe I/O.
>   3. git-upload-pack receives arguments such as 'depth' to fetch commits,
>      and send back to the caller.
>
> Therefore, if we want to implement "--deepen" for "git fetch", we have to
> modify the arguments protocol of git-upload-pack service by adding the
> 'shallow commit' hash.
> Then, we can fetch `depth` commits before the 'shallow commit' in
> git-upload-pack service.
> Apparently, we have to output error message when the 'shallow commit' hash
> is not in the repository.

Close. You can't figure this shallow commit hash from client side
(client repo is shortened). So you can't send it. What you send is
exactly what the user gives you (e.g. --deepen=5, then you send
"deepen 5" or similar). The server side (git-upload-pack) knows about
the shallow boundary of the client and can walk its (i.e. server-side)
commit graph to find out the new, deepened boundary. Then the server
sends necessary objects plus instructions to update shallow boundary
to the client.

Also, I think  this work would include support for smart-http
protocol. It goes a slightly different route, "git fetch" ends up at
transport.c, but then transport-helper.c and calls git-remote-http(s)
which is implemented by remote-curl.c. This one handles http stuff
then passes control to git-fetch-pack.c. It's fetch-pack that talks to
upload-pack.c

Have a look at Documentation/technical/pack-protocol.txt and
protocol-capabilities.txt (and http-protocol.txt for smart-http). The
function that "draws" shallow boundary based on --depth is
get_shallow_commits() in shallow.c. I suspect you need to improve it a
bit to use with --deepen. I guess you can look at these [1]. Those
changes touch shallow repo in a bit different way, but the main code
path is more or less the same (smart-http not touched).

[1] https://github.com/pclouds/git/commits/7edde8b83a20abb3cd404e2c5f07e3c29a2891f7
-- 
Duy

Re: [GSoC/RFC] Ideas on git fetch --deepen

[Dongcan Jiang]

It is really helpful. Thanks a lot!

2015-03-08 17:34 GMT+08:00 Duy Nguyen <pclouds@gmail.com>:
> On Sun, Mar 8, 2015 at 3:57 PM, Dongcan Jiang <dongcan.jiang@gmail.com> wrote:
>> Hi, all
>>
>> After digging into how "git fetch" works, I have found that my previous
>> understanding was too rash. I'm sorry for that.
>>
>> I find that the current workflow of "git fetch --depth" is as follows:
>>
>>   1. 'fetch module' calls 'git-upload-pack service' via 'transport module'
>>      to fetch ref with `depth`.
>>   2. Such call is finished by pipe I/O.
>>   3. git-upload-pack receives arguments such as 'depth' to fetch commits,
>>      and send back to the caller.
>>
>> Therefore, if we want to implement "--deepen" for "git fetch", we have to
>> modify the arguments protocol of git-upload-pack service by adding the
>> 'shallow commit' hash.
>> Then, we can fetch `depth` commits before the 'shallow commit' in
>> git-upload-pack service.
>> Apparently, we have to output error message when the 'shallow commit' hash
>> is not in the repository.
>
> Close. You can't figure this shallow commit hash from client side
> (client repo is shortened). So you can't send it. What you send is
> exactly what the user gives you (e.g. --deepen=5, then you send
> "deepen 5" or similar). The server side (git-upload-pack) knows about
> the shallow boundary of the client and can walk its (i.e. server-side)
> commit graph to find out the new, deepened boundary. Then the server
> sends necessary objects plus instructions to update shallow boundary
> to the client.
>
> Also, I think  this work would include support for smart-http
> protocol. It goes a slightly different route, "git fetch" ends up at
> transport.c, but then transport-helper.c and calls git-remote-http(s)
> which is implemented by remote-curl.c. This one handles http stuff
> then passes control to git-fetch-pack.c. It's fetch-pack that talks to
> upload-pack.c
>
> Have a look at Documentation/technical/pack-protocol.txt and
> protocol-capabilities.txt (and http-protocol.txt for smart-http). The
> function that "draws" shallow boundary based on --depth is
> get_shallow_commits() in shallow.c. I suspect you need to improve it a
> bit to use with --deepen. I guess you can look at these [1]. Those
> changes touch shallow repo in a bit different way, but the main code
> path is more or less the same (smart-http not touched).
>
> [1] https://github.com/pclouds/git/commits/7edde8b83a20abb3cd404e2c5f07e3c29a2891f7
> --
> Duy



-- 
江东灿（Dongcan Jiang）
Team of Search Engine & Web Mining
School of Electronic Engineering & Computer Science
Peking University, Beijing, 100871, P.R.China