New release?

New release?

[Junio C Hamano]

The "next" queue has been shrinking, and nothing is going to
"maint" branch, which can mean only one thing ;-).

The last issue of "What's in" talked about 1.4.0-rc1 but somehow
vger threw it on the bitbucket.  The message started like this:

        Subject: What's in git.git
        To: git@vger.kernel.org
	Date: Wed, 31 May 2006 23:40:26 -0700

        It's been a while since the last feature release, and I
        think with the recent "many things built-in" (including the
        busybox style integration) we are nearing a good time to do the
        next feature release 1.4.0.

        Before doing a 1.4.0-rc1, I would like to see the following
        topics in the "next" branch graduate to "master":

         - re-add missing flags to format-patch.  I have resurrected
           "--signoff"; if people care about something else we dropped
           when we went built-in, please raise hand and submit patches.

         - tree-parser updates from Linus seems to be fine in the sense
           I haven't seen breakage from it; I'll push it out to "master"
           before the end of the week.  I'd like to do another round of
           update to introduce a unified tree/index/directory walker, so
           settling this down is sort of urgent.

         - http-fetch fixes from Nick, which looked obviously correct.
           I would appreciate test reports from people who saw breakages
           on this one.

         - reflog from Shawn.  Do people find this useful?  I've enabled
           reflog on "next" branch in my development repository to see
           how useful it would be for myself a few days ago, and also in
           a linux-2.6 repository I use for testing (I do not hack on
           kernel myself).  

All of the above have been cleared with the recent fixes to
miscellaneous minor breakages around the tree-parser updates, so
I think what we have on the "master" branch tonight is worthy of
1.4.0-rc1 status.

        Other topics in "next" includes:

         - read/write-tree --prefix.  This is remnant of now-vetoed
           subproject support using "bind commit".  I kept both of them
           because they could be useful independent of "bind commit",
           but I do not know how much.  I think read-tree --prefix might
           probably be more useful than write-tree --prefix, since the
           latter can be writing out the whole tree and run rev-parse
           $tree:/path/name to extract that part, but the former does
           not have an easy equivalent (you could pipe ls-tree output to
           sed and pipe that to update-cache --index-info, but that is
           crumsy). 

           I'd like to do "gitlink" based subproject support but most
           likely that needs to come after tree/index/directory walker.

I am inclined to drop "read/write-tree --prefix" even from
"next"; we can resurrect it later if it proves to be useful in
order to do something else, such as "gitlink".

         - fetch-pack client-side hack.  When your repository has more
           roots than the repository you are fetching from, the common
           commit discovery exchange between fetch-pack and upload-pack
           ends up traversing down the ancestry chain of the history the
           other end do not have.  The hack in the "next" branch is to
           give up the common commit discovery early on the client side,
           which Ralf Baechle who originally reported the problem says
           to fix the problem (<20060526154239.GA20839@linux-mips.org>);
           but the proper fix involves a bit smarter upload-pack.

           I've posted a hacky upload-pack patch:

                <7vfyiwi4xl.fsf@assigned-by-dhcp.cox.net>

           but I think it should really needs to be cleaned up properly.

I think this can happen after 1.4.0.

        Things that we might want to have in 1.4.0 but not even in "next"
        yet include:

         - p4 importer (Sean Estabrooks) -- are people interested?

I am in favour of doing this before 1.4.0 (not in contrib/ but
as an importer next to other git-*import).

         - letting fetch-pack ask for an arbitrary commit object the
           user obtained out of band (Eric W Biederman) -- waiting for
           updated patch.  We would need a corresponding one-liner patch
           to upload-pack when we do this.

This can wait.

         - using ~/.gitrc to give a fall-back default when
           $GIT_DIR/config does not have values.

I suspect this would be more involved than Pasky's initial
patch; but it can wait.

         - command aliases and possibly default arguments via the
           configuration file.

This we have in "next" tonight and we should be able to have it
in "master" by the end of the week.  I took what Johannes did,
but I think that is pretty much the same as what Pasky did
initially, but forgot to mention that in the commit log.

I am expected to be mostly offline next week (among other
things, http://osdl.jp/seminar0613/ if you can read Japanese),
so I am planning to do 1.4.0-rc1 by my next git day (Wednesday
7th), and perhaps the real 1.4.0 on Saturday 10th if things go
smoothly.

Now v1.4.0-rc1 seems to have mirrored out.  Please have fun.

Re: New release?

[Johannes Schindelin]

Hi,

On Mon, 5 Jun 2006, Junio C Hamano wrote:

>          - letting fetch-pack ask for an arbitrary commit object the
>            user obtained out of band (Eric W Biederman) -- waiting for
>            updated patch.  We would need a corresponding one-liner patch
>            to upload-pack when we do this.
> 
> This can wait.

I think that this could be an important step towards a sensible "shallow 
clone": we could introduce "remote alternates", where sha1_file 
transparently loads (and stores) single commit objects and tree objects 
together with their subtrees and blobs from a remote, if they are not 
present in the shallow repo yet.

This will be blazingly _slow_, I guess, if one does any such stupid thing 
as git-fsck-objects... However, it should be fixable (expensive commands, 
i.e. commands traversing extensive history/object lists, could just barf 
if the repo is shallow).

>          - using ~/.gitrc to give a fall-back default when
>            $GIT_DIR/config does not have values.
> 
> I suspect this would be more involved than Pasky's initial
> patch; but it can wait.

I think that this is quite important for the aliases to be useful. 
However, it does not blend well with the mmap()ifying of the config 
reading.

Also, there is the problem of unique keys. You want unique keys to be only 
unique _per config file_, i.e. the local config can override the user's 
config. This is probably only relevant for git-repo-config, though.

Ciao,
Dscho

Re: New release?

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Mon, 5 Jun 2006, Junio C Hamano wrote:
>
>>          - letting fetch-pack ask for an arbitrary commit object the
>>            user obtained out of band (Eric W Biederman) -- waiting for
>>            updated patch.  We would need a corresponding one-liner patch
>>            to upload-pack when we do this.
>> 
>> This can wait.
>
> I think that this could be an important step towards a sensible "shallow 
> clone"...

I did not say we are not interested in doing this ever.  The
"arbitrary commit" thing is easy but I do not think it is all
that important to hold all the good stuff back that happened
since 1.3.0 and delay 1.4.0.

Also, what you talk about the "lazy clone" is a lot more
involved than what Eric wanted to have.  It is _never_ shallow
clones that normal users would want -- making it shallow to
explicitly say certain operations break is a cop-out for
implementors.  What the users really want is to be in total
control -- ranging from completely on-demand ala CVS and SVN to
"down to this commit in the history I would want to be cached on
the local machine so that I can go offline and still do useful
things with the history", with new failure modes for history
traversing commands to exit gracefully when offline.  That _is_
the ideal but I know it won't be within reach anytime soon.

>>          - using ~/.gitrc to give a fall-back default when
>>            $GIT_DIR/config does not have values.
>> 
>> I suspect this would be more involved than Pasky's initial
>> patch; but it can wait.
>
> I think that this is quite important for the aliases to be useful. 

I am not saying this is unimportant.  Again, however, it is not
that important to hold other good stuff.

Re: New release?

[Johannes Schindelin]

Hi,

On Tue, 6 Jun 2006, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > On Mon, 5 Jun 2006, Junio C Hamano wrote:
> >
> >>          - letting fetch-pack ask for an arbitrary commit object the
> >>            user obtained out of band (Eric W Biederman) -- waiting for
> >>            updated patch.  We would need a corresponding one-liner patch
> >>            to upload-pack when we do this.
> >> 
> >> This can wait.
> >
> > I think that this could be an important step towards a sensible "shallow 
> > clone"...
> 
> I did not say we are not interested in doing this ever.

I was not trying to make a case to wait with 1.4.0...

> Also, what you talk about the "lazy clone" is a lot more involved than 
> what Eric wanted to have.

A little more involved. And I wanted to know what people think about this 
way to tackle shallow clones.

Ciao,
Dscho

Re: New release?

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> Also, what you talk about the "lazy clone" is a lot more involved than 
>> what Eric wanted to have.
>
> A little more involved. And I wanted to know what people think about this 
> way to tackle shallow clones.

This truly is a lot more involved, not because "lazy clone" is
hard (which it is) but because what Eric wants to have is not to
produce a shallow clone situation.  The issue is about being
able to fetch an object that is not listed as one of the refs
the server side has, and it still will result in a fully
connected repository.

Re: New release?

[Jon Loeliger]

On Tue, 2006-06-06 at 01:02, Junio C Hamano wrote:
> The "next" queue has been shrinking, and nothing is going to
> "maint" branch, which can mean only one thing ;-).

Junio,

If possible, I'd like to consider my (Linus') notion
of the extension to the git protocol supplying the client
hostname as part of a "standard" release like 1.4.0.

We don't necessarily have to agree on how it is used yet,
but if we can get the protocol enhancement into this
release, I think it would make for a good "flag day"
sort of change.

Specifically, I'd like to consider the portion my patch
that passes "\0HOST=%s\0" along with the git: protocol
connection.

Any chance for that?  If you'd like, I work on resubmitting
just those bits with the connect function refactoring
that you outlined.

Thanks,
jdl

Re: New release?

[Junio C Hamano]

Jon Loeliger <jdl@freescale.com> writes:

> If possible, I'd like to consider my (Linus') notion
> of the extension to the git protocol supplying the client
> hostname as part of a "standard" release like 1.4.0.

I think the virtual hosting of git-daemon is very important, and
I do not want to have a half-baked one hurriedly cobbled
together for 1.4.0 on the server side.

> We don't necessarily have to agree on how it is used yet,
> but if we can get the protocol enhancement into this
> release, I think it would make for a good "flag day"
> sort of change.

I do agree it is a good idea to have the client-side support in
1.4.0; that would work well with the current git-daemon.  Since
the extension is backward compatible, we can tell the users to
use 1.4.0 or later clients when the daemon side is done.  That
is much nicer than telling them to use what's on "master" that
is later than such-and-such commit.

> Specifically, I'd like to consider the portion my patch
> that passes "\0HOST=%s\0" along with the git: protocol
> connection.
>
> Any chance for that?  If you'd like, I work on resubmitting
> just those bits with the connect function refactoring
> that you outlined.

I appreciate the offer very much.  Although I said port is
something the server side can usually tell, I would throw in
port there as well.  Maybe there is some port forwarding (or
reverse NAPT) involved in the path from the client to the
server.  Any reason to spell HOST in all uppercase by the way?

And thanks for reminding me.  The list of things-to-have in the
message was based on a week old "What's in", and I was about to
compose a message describing what I did not talk about in it,
because I wanted to say something about virtual hosting of
git-daemon.

Re: New release?

[Jon Loeliger]

On Tue, 2006-06-06 at 12:15, Junio C Hamano wrote:

> I think the virtual hosting of git-daemon is very important, and
> I do not want to have a half-baked one hurriedly cobbled
> together for 1.4.0 on the server side.

Agreed.

> I do agree it is a good idea to have the client-side support in
> 1.4.0; that would work well with the current git-daemon.  Since
> the extension is backward compatible, we can tell the users to
> use 1.4.0 or later clients when the daemon side is done.  That
> is much nicer than telling them to use what's on "master" that
> is later than such-and-such commit.

Exactly.

> > Any chance for that?  If you'd like, I work on resubmitting
> > just those bits with the connect function refactoring
> > that you outlined.
> 
> I appreciate the offer very much.

No problem.

>   Although I said port is
> something the server side can usually tell, I would throw in
> port there as well.

Can do.

>  Any reason to spell HOST in all uppercase by the way?

I would never suggest that it came to me from 
gittus in an IRC day-dream...? :-)

Consider it all lowercaseified instead.

jdl

Re: New release?

[Pavel Roskin]

Hello, Junio!

On Mon, 2006-06-05 at 23:02 -0700, Junio C Hamano wrote:
>          - http-fetch fixes from Nick, which looked obviously correct.
>            I would appreciate test reports from people who saw breakages
>            on this one.

I'm still getting a segfault with the current git from the "next" branch:

$ git-clone http://www.denx.de/git/linux-2.6-denx.git
...
got 4160b8334c53e0881cdc12c1f7d3d54fff883772
got 5f57f29efee48d84e235a8ff75a35e7e354227a7
got 681a9c73a2a321850404d4856f4738be47e17d15
got 29b0ddaa324e417abf153460d7d94fb67823a6ef
got 23e7a5c7d2c13d98524b69f54378d887e1962fc8
/home/proski/bin/git-clone: line 29: 27271 Segmentation fault      git-http-fetch -v -a -w "$tname" "$name" "$1/"

It takes about an hour with my connection from running the command to
the segfault.  It you have any idea how to speed it up, it would be very
helpful.

If I comment out USE_CURL_MULTI, I get compile errors in http.c and
http-push.c.

http.c: In function 'release_active_slot':
http.c:441: error: 'curlm' undeclared (first use in this function)
http.c:441: error: (Each undeclared identifier is reported only once
http.c:441: error: for each function it appears in.)
http.c:445: warning: implicit declaration of function
'fill_active_slots'

http-push.c: In function 'fill_active_slots':
http-push.c:801: error: 'max_requests' undeclared (first use in this
function)
http-push.c:801: error: (Each undeclared identifier is reported only
once
http-push.c:801: error: for each function it appears in.)
http-push.c:811: error: 'curlm' undeclared (first use in this function)
http-push.c: In function 'add_fetch_request':
http-push.c:855: warning: implicit declaration of function
'step_active_slots'

It may be easy to fix for somebody familiar with the code, but I don't
want to do it blindly.

-- 
Regards,
Pavel Roskin

Re: New release?

[Nick Hengeveld]

On Tue, Jun 06, 2006 at 12:19:19PM -0400, Pavel Roskin wrote:

> On Mon, 2006-06-05 at 23:02 -0700, Junio C Hamano wrote:
> >          - http-fetch fixes from Nick, which looked obviously correct.
> >            I would appreciate test reports from people who saw breakages
> >            on this one.
> 
> I'm still getting a segfault with the current git from the "next" branch:
> 
> $ git-clone http://www.denx.de/git/linux-2.6-denx.git
> ...
> got 4160b8334c53e0881cdc12c1f7d3d54fff883772
> got 5f57f29efee48d84e235a8ff75a35e7e354227a7
> got 681a9c73a2a321850404d4856f4738be47e17d15
> got 29b0ddaa324e417abf153460d7d94fb67823a6ef
> got 23e7a5c7d2c13d98524b69f54378d887e1962fc8
> /home/proski/bin/git-clone: line 29: 27271 Segmentation fault      git-http-fetch -v -a -w "$tname" "$name" "$1/"
> 
> It takes about an hour with my connection from running the command to
> the segfault.  It you have any idea how to speed it up, it would be very
> helpful.

I can't offer any suggestions wrt speeding it up, but I'll do what I can
to reproduce the problem here.

> If I comment out USE_CURL_MULTI, I get compile errors in http.c and
> http-push.c.

I just posted a fix for the compile errors.  As noted there, I've done
very little testing of the fetch/push binaries when built with
USE_CURL_MULTI commented out.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: New release?

[Pavel Roskin]

On Tue, 2006-06-06 at 09:46 -0700, Nick Hengeveld wrote:
> On Tue, Jun 06, 2006 at 12:19:19PM -0400, Pavel Roskin wrote:
> 
> > On Mon, 2006-06-05 at 23:02 -0700, Junio C Hamano wrote:
> > >          - http-fetch fixes from Nick, which looked obviously correct.
> > >            I would appreciate test reports from people who saw breakages
> > >            on this one.
> > 
> > I'm still getting a segfault with the current git from the "next" branch:
> > 
> > $ git-clone http://www.denx.de/git/linux-2.6-denx.git
> > ...
> > got 4160b8334c53e0881cdc12c1f7d3d54fff883772
> > got 5f57f29efee48d84e235a8ff75a35e7e354227a7
> > got 681a9c73a2a321850404d4856f4738be47e17d15
> > got 29b0ddaa324e417abf153460d7d94fb67823a6ef
> > got 23e7a5c7d2c13d98524b69f54378d887e1962fc8
> > /home/proski/bin/git-clone: line 29: 27271 Segmentation fault      git-http-fetch -v -a -w "$tname" "$name" "$1/"

It crashed again, in a different place:
got d82e6dae84070951f625622229154cb32d3f2333
got c3e1d3e888d7b25c20d90ae4a7ecb8f5be420b98
got 2ed07112d683fa7cd1c72b0a31a7e95c6645543d
got 2721e4c8184a9a64f505686ebf7bf6e1e80ecf59

This time I was better prepared (git was compiled with -g without
optimization, the trap was commented out in git-clone, "ulimit -c" set
to unlimited), and I attached gdb to git-http-fetch.

Program received signal SIGSEGV, Segmentation fault.
0x00000000004044dc in closedown_active_slot (slot=0x656854202a200a73) at http.c:434
434             slot->in_use = 0;
(gdb) p slot
$1 = (struct active_request_slot *) 0x656854202a200a73
(gdb) p slot->in_use
Cannot access memory at address 0x656854202a200a83
(gdb) where
#0  0x00000000004044dc in closedown_active_slot (slot=0x656854202a200a73) at http.c:434
#1  0x00000000004044fa in release_active_slot (slot=0x656854202a200a73) at http.c:439
#2  0x00000000004070fc in abort_object_request (obj_req=0xdaf2b0) at http-fetch.c:1060
#3  0x00000000004071cf in fetch_object (repo=0x548f50, 
    sha1=0xcd4838 "ñ¿¥\025×Ûþ¥c'\210æË©\213Ö}Ûôü") at http-fetch.c:1078
#4  0x00000000004073ed in fetch (sha1=0xcd4838 "ñ¿¥\025×Ûþ¥c'\210æË©\213Ö}Ûôü")
    at http-fetch.c:1126
#5  0x0000000000403126 in loop () at fetch.c:180
#6  0x000000000040336a in pull (target=0x7fff0c2e38c2 "heads/master") at fetch.c:248
#7  0x0000000000407a14 in main (argc=7, argv=0x7fff0c2e18a8) at http-fetch.c:1271
(gdb)

It's a different backtrace this time.  abort_object_request() has this code:

if (obj_req->slot) {
     release_active_slot(obj_req->slot);
     obj_req->slot = NULL;
}

Apparently just because obj_req->slot is not NULL doesn't mean it's a
valid pointer.  I'm going to use Valgrind now.

It's x86_64, FC5, Linux kernel from git.

> I just posted a fix for the compile errors.  As noted there, I've done
> very little testing of the fetch/push binaries when built with
> USE_CURL_MULTI commented out.

Thank you.  I'll try it too.

-- 
Regards,
Pavel Roskin

http-fetch segfault fix?

[Junio C Hamano]

Pavel Roskin <proski@gnu.org> writes:

> It's a different backtrace this time.  abort_object_request() has this code:
>
> if (obj_req->slot) {
>      release_active_slot(obj_req->slot);
>      obj_req->slot = NULL;
> }
>
> Apparently just because obj_req->slot is not NULL doesn't mean it's a
> valid pointer.  I'm going to use Valgrind now.

Nick's one-liner to explicitly initialize newreq->slot to NULL
looks obviously correct to me.  Does it fix this problem for
you?

Re: http-fetch segfault fix?

[Pavel Roskin]

On Tue, 2006-06-06 at 22:35 -0700, Junio C Hamano wrote:
> Pavel Roskin <proski@gnu.org> writes:
> 
> > It's a different backtrace this time.  abort_object_request() has this code:
> >
> > if (obj_req->slot) {
> >      release_active_slot(obj_req->slot);
> >      obj_req->slot = NULL;
> > }
> >
> > Apparently just because obj_req->slot is not NULL doesn't mean it's a
> > valid pointer.  I'm going to use Valgrind now.
> 
> Nick's one-liner to explicitly initialize newreq->slot to NULL
> looks obviously correct to me.  Does it fix this problem for
> you?

I'm going to leave the tests running overnight, both with and without
USE_CURL_MULTI.

The Valgrind diagnostics confirms that obj_req->slot is not initialized
(as opposed to being a pointer to a freed area or something else):

==27182== Conditional jump or move depends on uninitialised value(s)
==27182==    at 0x4070EA: abort_object_request (http-fetch.c:1059)
==27182==    by 0x4071CE: fetch_object (http-fetch.c:1078)
==27182==    by 0x4073EC: fetch (http-fetch.c:1126)
==27182==    by 0x403125: loop (fetch.c:180)
==27182==    by 0x403369: pull (fetch.c:248)
==27182==    by 0x407A13: main (http-fetch.c:1271)

Line 1059 is:
if (obj_req->slot) {

-- 
Regards,
Pavel Roskin

Re: http-fetch segfault fix?

[Junio C Hamano]

Pavel Roskin <proski@gnu.org> writes:

> The Valgrind diagnostics confirms that obj_req->slot is not initialized
> (as opposed to being a pointer to a freed area or something else):
>
> ==27182== Conditional jump or move depends on uninitialised value(s)
> ==27182==    at 0x4070EA: abort_object_request (http-fetch.c:1059)
> ==27182==    by 0x4071CE: fetch_object (http-fetch.c:1078)
> ==27182==    by 0x4073EC: fetch (http-fetch.c:1126)
> ==27182==    by 0x403125: loop (fetch.c:180)
> ==27182==    by 0x403369: pull (fetch.c:248)
> ==27182==    by 0x407A13: main (http-fetch.c:1271)
>
> Line 1059 is:
> if (obj_req->slot) {

Thanks.  That is indeed a very good sign.

Re: http-fetch segfault fix?

[Pavel Roskin]

On Tue, 2006-06-06 at 22:58 -0700, Junio C Hamano wrote:
> Pavel Roskin <proski@gnu.org> writes:
> 
> > The Valgrind diagnostics confirms that obj_req->slot is not initialized
> > (as opposed to being a pointer to a freed area or something else):
> >
> > ==27182== Conditional jump or move depends on uninitialised value(s)
> > ==27182==    at 0x4070EA: abort_object_request (http-fetch.c:1059)
> > ==27182==    by 0x4071CE: fetch_object (http-fetch.c:1078)
> > ==27182==    by 0x4073EC: fetch (http-fetch.c:1126)
> > ==27182==    by 0x403125: loop (fetch.c:180)
> > ==27182==    by 0x403369: pull (fetch.c:248)
> > ==27182==    by 0x407A13: main (http-fetch.c:1271)
> >
> > Line 1059 is:
> > if (obj_req->slot) {
> 
> Thanks.  That is indeed a very good sign.

Both git-clone instances (with and without USE_CURL_MULTI) have
completed successfully on http://www.denx.de/git/linux-2.6-denx.git

Nick, thank you for fixing this bug!

-- 
Regards,
Pavel Roskin

Re: http-fetch segfault fix?

[Nick Hengeveld]

On Wed, Jun 07, 2006 at 10:29:44AM -0400, Pavel Roskin wrote:

> Both git-clone instances (with and without USE_CURL_MULTI) have
> completed successfully on http://www.denx.de/git/linux-2.6-denx.git
> 
> Nick, thank you for fixing this bug!

Whew...  I look forward to working on something other than fixing
segfaults.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.