GSoC proposal: port pack bitmap support to libgit2.

GSoC proposal: port pack bitmap support to libgit2.

[Yuxuan Shui]

Hi,

I'm Yuxuan Shui, a undergraduate student from China. I'm applying for
GSoC 2014, and here is my proposal:

I found this idea on the ideas page, and did some research about it.
The pack bitmap patchset add a new .bitmap file for every pack file
which contains the reachability information of selected commits. This
information is used to speed up git fetching and cloning, and produce
a very convincing results.

The goal of my project is to port the pack bitmap implementation in
core git to libgit2, so users of libgit2 could benefit from this
optimization as well.

Please let me know if my proposal makes sense, thanks.

P.S. I've submitted by microproject patch[1], but haven't received any
response yet.

[1]: http://thread.gmane.org/gmane.comp.version-control.git/243854

-- 
Regards
Yuxuan Shui

Re: GSoC proposal: port pack bitmap support to libgit2.

[Yuxuan Shui]

Hi,

On Wed, Mar 12, 2014 at 4:19 PM, Yuxuan Shui <yshuiv7@gmail.com> wrote:
> Hi,
>
> I'm Yuxuan Shui, a undergraduate student from China. I'm applying for
> GSoC 2014, and here is my proposal:
>
> I found this idea on the ideas page, and did some research about it.
> The pack bitmap patchset add a new .bitmap file for every pack file
> which contains the reachability information of selected commits. This
> information is used to speed up git fetching and cloning, and produce
> a very convincing results.
>
> The goal of my project is to port the pack bitmap implementation in
> core git to libgit2, so users of libgit2 could benefit from this
> optimization as well.
>
> Please let me know if my proposal makes sense, thanks.
>
> P.S. I've submitted by microproject patch[1], but haven't received any
> response yet.
>
> [1]: http://thread.gmane.org/gmane.comp.version-control.git/243854
>
> --
> Regards
> Yuxuan Shui

Could anyone please review my proposal a little bit? Is this project
helpful and worth doing? Did I get anything wrong in my proposal?

Thanks.

-- 

Regards
Yuxuan Shui

Re: GSoC proposal: port pack bitmap support to libgit2.

[Jeff King]

On Wed, Mar 12, 2014 at 04:19:23PM +0800, Yuxuan Shui wrote:

> I'm Yuxuan Shui, a undergraduate student from China. I'm applying for
> GSoC 2014, and here is my proposal:
> 
> I found this idea on the ideas page, and did some research about it.
> The pack bitmap patchset add a new .bitmap file for every pack file
> which contains the reachability information of selected commits. This
> information is used to speed up git fetching and cloning, and produce
> a very convincing results.
> 
> The goal of my project is to port the pack bitmap implementation in
> core git to libgit2, so users of libgit2 could benefit from this
> optimization as well.
> 
> Please let me know if my proposal makes sense, thanks.

You'd want to flesh it out a bit more to show how you're thinking about
tackling the problem:

  - What are the areas of libgit2 that you will need to touch? Be
    specific. What's the current state of the packing code? What
    files and functions will you need to touch?

  - What are the challenges you expect to encounter in porting the code?

  - Can you give a detailed schedule of the summer's work? What will you
    work on in each week? What milestones do you expect to hit, and
    when?

-Peff

Re: GSoC proposal: port pack bitmap support to libgit2.

[Yuxuan Shui]

Hi,

Sorry for this late reply, I was busy for past few days.

On Fri, Mar 14, 2014 at 12:34 PM, Jeff King <peff@peff.net> wrote:
> On Wed, Mar 12, 2014 at 04:19:23PM +0800, Yuxuan Shui wrote:
>
>> I'm Yuxuan Shui, a undergraduate student from China. I'm applying for
>> GSoC 2014, and here is my proposal:
>>
>> I found this idea on the ideas page, and did some research about it.
>> The pack bitmap patchset add a new .bitmap file for every pack file
>> which contains the reachability information of selected commits. This
>> information is used to speed up git fetching and cloning, and produce
>> a very convincing results.
>>
>> The goal of my project is to port the pack bitmap implementation in
>> core git to libgit2, so users of libgit2 could benefit from this
>> optimization as well.
>>
>> Please let me know if my proposal makes sense, thanks.
>
> You'd want to flesh it out a bit more to show how you're thinking about
> tackling the problem:
>
>   - What are the areas of libgit2 that you will need to touch? Be
>     specific. What's the current state of the packing code? What
>     files and functions will you need to touch?

Firstly I will need to implement bitmap creation in libgit2's
git_packbuilder_* functions (probably also git_odb_write_pack), so
libgit2 could support bitmap creation. Then I will need to change
git_revwalk_* functions to make them use bitmap. Since the operations
that can benefit from bitmap is, if my understanding is correct, all
using the git_revwalk_* functions, having bitmap support in revwalk
functions should be enough.

Files I need to touch probably are: revwalk.c pack-objects.c
If I need to change the API of packbuilder or revwalk functions I will
have to change the callers as well: push.c fetch.c and
transport/smart_protocol.c

I haven't read all the code to put together a list of functions I need
to change, but I think the list will be long.

>
>   - What are the challenges you expect to encounter in porting the code?

The architecture differences between git and libgit2 will probably be
a challenge.

>
>   - Can you give a detailed schedule of the summer's work? What will you
>     work on in each week? What milestones do you expect to hit, and
>     when?

I don't really have a plan, but I'll try to provide a rough schedule.

I'll read the code and try to understand the code, to the point where
I can start to add new code. This will probably take a week. For next
three or four weeks I should be implementing bitmap creation in
packbuilder. Then for the rest of time I will be optimizing revwalk
using bitmap.

>
> -Peff

--

Regards
Yuxuan Shui