Trying to understand the web dav details

Trying to understand the web dav details

[Jason Pyeron]

When doing a clone by https (reverse proxied to http) the first request is

GET /git/project/info/refs?service=git-upload-pack

How does the ?service=xxxx get translated in to the action performed on the web
server?

I ask because I have 2 projects, one works the other does not.

I am using httpd-2.0.52-49.ent.centos4 and git-1.7.9.6-1.

I am not even sure what to tell more about or where to look next.

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Re: Trying to understand the web dav details

[Jeff King]

On Sat, Jan 05, 2013 at 08:32:09PM -0500, Jason Pyeron wrote:

> When doing a clone by https (reverse proxied to http) the first request is
> 
> GET /git/project/info/refs?service=git-upload-pack
> 
> How does the ?service=xxxx get translated in to the action performed on the web
> server?

If you are using the git-http-backend CGI, it will interpret the service
tag and start smart-http. See "git help http-backend" for details on
plugging it into Apache.

Cloning/fetching does not use DAV at all; it is only for non-smart
pushing (and I would not recommend setting it up; the smart protocol
spoken by git-http-backend does pushing much more efficiently, and is
better maintained).

> I ask because I have 2 projects, one works the other does not.
> 
> I am using httpd-2.0.52-49.ent.centos4 and git-1.7.9.6-1.
> 
> I am not even sure what to tell more about or where to look next.

If you haven't set up git-http-backend, then git is just going to fetch
the remote repo's data directly over http. So the usual advice for
accessing something via http would apply (check the server's access and
error logs, try hitting it with a web browser, etc).

If you set GIT_CURL_VERBOSE=1 in your environment, git will spew a lot
of debugging information about what http requests it is making. That
might give you a clue (you haven't said anything about what does not
work, so I can't be more specific).

-Peff

RE: Trying to understand the web dav details

[Pyeron, Jason J CTR (US)]

[-- Attachment #1: Type: text/plain, Size: 11629 bytes --]

> -----Original Message-----
> From: Jeff King
> Sent: Saturday, January 05, 2013 11:20 PM
> 
> On Sat, Jan 05, 2013 at 08:32:09PM -0500, Jason Pyeron wrote:
> 
> > When doing a clone by https (reverse proxied to http) the first
> request is
> >
> > GET /git/project/info/refs?service=git-upload-pack
> >
> > How does the ?service=xxxx get translated in to the action performed
> on the web
> > server?
> 
> If you are using the git-http-backend CGI, it will interpret the
> service

No, using plain jane http and webdav. This server is not "allowed" to use cgi processes.

> tag and start smart-http. See "git help http-backend" for details on
> plugging it into Apache.
> 
> Cloning/fetching does not use DAV at all; it is only for non-smart
> pushing (and I would not recommend setting it up; the smart protocol
> spoken by git-http-backend does pushing much more efficiently, and is
> better maintained).
> 
> > I ask because I have 2 projects, one works the other does not.

Should have said I get a 404 versus 200, see below.

> >
> > I am using httpd-2.0.52-49.ent.centos4 and git-1.7.9.6-1.
> >
> > I am not even sure what to tell more about or where to look next.
> 
> If you haven't set up git-http-backend, then git is just going to fetch
> the remote repo's data directly over http. So the usual advice for
> accessing something via http would apply (check the server's access and
> error logs, try hitting it with a web browser, etc).
> 
> If you set GIT_CURL_VERBOSE=1 in your environment, git will spew a lot

That’s what I did before mailing out the first time. I tried to summarize the contents, but I will paste the relevant portions below.

> of debugging information about what http requests it is making. That
> might give you a clue (you haven't said anything about what does not
> work, so I can't be more specific).

Below are two traces, the first one is the one that does not work, the second does. I cannot see any differences on the web server for the two directory trees.

**************************************************************************
**************************************************************************
**************************************************************************
**************************************************************************

jason.pyeron@localhost ~/tmp
$ GIT_TRACE=1 GIT_CURL_VERBOSE=1 git clone https://server/git/project-x
trace: built-in: git 'clone' 'https://server/git/project-x'
Cloning into 'project-x'...
trace: run_command: 'git-remote-https' 'origin' 'https://server/git/project-x'
* Couldn't find host server in the .netrc file; using defaults
* About to connect() to proxy 214.36.0.135 port 8080 (#0)
*   Trying 214.36.0.135...
* 0x80076a48 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x8007f3c0; (connection #0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Establish HTTP proxy tunnel to server:443
> CONNECT server:443 HTTP/1.1
Host: server:443
User-Agent: git/1.7.9
Proxy-Connection: Keep-Alive
Pragma: no-cache

* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x8007f3c0; (connection #0)
* Multi mode finished polling for response from proxy CONNECT
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* STATE: WAITPROXYCONNECT => WAITCONNECT handle 0x8007f3c0; (connection #0)
* STATE: WAITCONNECT => PROTOCONNECT handle 0x8007f3c0; (connection #0)
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: C=US; ST=Maryland; O=PD Inc; OU=Intranet; CN=server; emailAddress=security@pdinc.us
*        start date: 201
*        expire date: 201
*        issuer: C=U
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x8007f3c0; (connection #0)
> GET /git/project-x/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.9
Host: server
Accept: */*
Pragma: no-cache

* STATE: DO => DO_DONE handle 0x8007f3c0; (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x8007f3c0; (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x8007f3c0; (connection #0)
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* The requested URL returned error: 404 Not Found
* Closing connection #0
* Couldn't find host server in the .netrc file; using defaults
* About to connect() to proxy 214.36.0.135 port 8080 (#0)
*   Trying 214.36.0.135...
* 0x80076a48 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x80139640; (connection #0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Establish HTTP proxy tunnel to server:443
> CONNECT server:443 HTTP/1.1
Host: server:443
User-Agent: git/1.7.9
Proxy-Connection: Keep-Alive
Pragma: no-cache

* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x80139640; (connection #0)
* Multi mode finished polling for response from proxy CONNECT
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* SSL re-using session ID
* STATE: WAITPROXYCONNECT => WAITCONNECT handle 0x80139640; (connection #0)
* STATE: WAITCONNECT => PROTOCONNECT handle 0x80139640; (connection #0)
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: C=US; ST=Maryland; O=PD Inc; OU=Intranet; CN=server; emailAddress=security@pdinc.us
*        start date: 201
*        expire date: 201
*        issuer: C=U
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x80139640; (connection #0)
> GET /git/project-x/info/refs HTTP/1.1
User-Agent: git/1.7.9
Host: server
Accept: */*
Pragma: no-cache

* STATE: DO => DO_DONE handle 0x80139640; (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x80139640; (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x80139640; (connection #0)
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* The requested URL returned error: 404 Not Found
* Closing connection #0
fatal: https://server/git/project-x/info/refs not found: did you run git update-server-info on the server?

**************************************************************************
**************************************************************************
**************************************************************************
**************************************************************************

jason.pyeron@localhost ~/tmp
$ GIT_TRACE=1 GIT_CURL_VERBOSE=1 git clone https://server/git/test
trace: built-in: git 'clone' 'https://server/git/test'
Cloning into 'test'...
trace: run_command: 'git-remote-https' 'origin' 'https://server/git/test'
* Couldn't find host server in the .netrc file; using defaults
* About to connect() to proxy 214.36.0.135 port 8080 (#0)
*   Trying 214.36.0.135...
* 0x80076968 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x8007f2c8; (connection #0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Establish HTTP proxy tunnel to server:443
> CONNECT server:443 HTTP/1.1
Host: server:443
User-Agent: git/1.7.9
Proxy-Connection: Keep-Alive
Pragma: no-cache

* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x8007f2c8; (connection #0)
* Multi mode finished polling for response from proxy CONNECT
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* STATE: WAITPROXYCONNECT => WAITCONNECT handle 0x8007f2c8; (connection #0)
* STATE: WAITCONNECT => PROTOCONNECT handle 0x8007f2c8; (connection #0)
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: C=US; ST=Maryland; O=PD Inc; OU=Intranet; CN=server; emailAddress=security@pdinc.us
*        start date: 201
*        expire date: 201
*        issuer: C=U
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x8007f2c8; (connection #0)
> GET /git/test/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.9
Host: server
Accept: */*
Pragma: no-cache

* STATE: DO => DO_DONE handle 0x8007f2c8; (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x8007f2c8; (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x8007f2c8; (connection #0)
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Date: Sun, 06 Jan 2013 04:26:14 GMT
< Server: Apache/2.0.52 (CentOS)
< Last-Modified: Fri, 02 Nov 2012 05:05:55 GMT
< ETag: "714064-3b-172852c0"
< Accept-Ranges: bytes
< Content-Length: 59
< Content-Type: text/plain; charset=UTF-8
< Connection: close
<
* STATE: PERFORM => DONE handle 0x8007f2c8; (connection #0)
* Closing connection #0
* Couldn't find host server in the .netrc file; using defaults
* About to connect() to proxy 214.36.0.135 port 8080 (#0)
*   Trying 214.36.0.135...
* 0x80076968 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x801395e8; (connection #0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Connected to 214.36.0.135 (214.36.0.135) port 8080 (#0)
* Establish HTTP proxy tunnel to server:443
> CONNECT server:443 HTTP/1.1
Host: server:443
User-Agent: git/1.7.9
Proxy-Connection: Keep-Alive
Pragma: no-cache

* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x801395e8; (connection #0)
* Multi mode finished polling for response from proxy CONNECT
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* SSL re-using session ID
* STATE: WAITPROXYCONNECT => WAITCONNECT handle 0x801395e8; (connection #0)
* STATE: WAITCONNECT => PROTOCONNECT handle 0x801395e8; (connection #0)
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: C=US; ST=Maryland; O=PD Inc; OU=Intranet; CN=server; emailAddress=security@pdinc.us
*        start date: 201
*        expire date: 201
*        issuer: C=U
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x801395e8; (connection #0)
> GET /git/test/HEAD HTTP/1.1

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5615 bytes --]

Re: Trying to understand the web dav details

[Jeff King]

On Sun, Jan 06, 2013 at 04:49:57AM +0000, Pyeron, Jason J CTR (US) wrote:

> > > How does the ?service=xxxx get translated in to the action
> > > performed on the web server?
> > 
> > If you are using the git-http-backend CGI, it will interpret the
> > service
> 
> No, using plain jane http and webdav. This server is not "allowed" to
> use cgi processes.

Then the service parameter should be ignored by your webserver, and it
should just serve the info/refs file from the repository on the
filesystem. And you are stuck using WebDAV for push.

> > GET /git/project-x/info/refs HTTP/1.1
> [...]
> * The requested URL returned error: 404 Not Found

Does the info/refs file exist in the project-x repository?

> fatal: https://server/git/project-x/info/refs not found: did you run git update-server-info on the server?

Did you?

If you can't run any git programs on the server at all (and it sounds
like that may be the case), you'll need to run it locally before putting
the repository data on the server.

Once you have WebDAV set up for pushing, it will update the info/refs
file for each push. But if you are initially seeding the server with
rsync or a tarfile, you'll want to make sure it has an up-to-date
info/refs file.

-Peff

RE: Trying to understand the web dav details

[Jason Pyeron]

Ignore everything below, it was a case sensitive typo. It always worked it.

> -----Original Message-----
> From: Jeff King
> Sent: Sunday, January 06, 2013 0:38
> 
> On Sun, Jan 06, 2013 at 04:49:57AM +0000, Pyeron, Jason J CTR 
> (US) wrote:
> 
> > > > How does the ?service=xxxx get translated in to the action 
> > > > performed on the web server?
> > > 
> > > If you are using the git-http-backend CGI, it will interpret the 
> > > service
> > 
> > No, using plain jane http and webdav. This server is not 
> "allowed" to 
> > use cgi processes.
> 
> Then the service parameter should be ignored by your 
> webserver, and it should just serve the info/refs file from 
> the repository on the filesystem. And you are stuck using 
> WebDAV for push.
> 
> > > GET /git/project-x/info/refs HTTP/1.1
> > [...]
> > * The requested URL returned error: 404 Not Found
> 
> Does the info/refs file exist in the project-x repository?

Yes.

> 
> > fatal: https://server/git/project-x/info/refs not found: 
> did you run git update-server-info on the server?
> 
> Did you?
> 

Many times.

> If you can't run any git programs on the server at all (and 
> it sounds like that may be the case), you'll need to run it 
> locally before putting the repository data on the server.
> 
> Once you have WebDAV set up for pushing, it will update the 
> info/refs file for each push. But if you are initially 
> seeding the server with rsync or a tarfile, you'll want to 

Seeding it seems to work, it is the bare init that seems to be failing. Might be
on to something there.

> make sure it has an up-to-date info/refs file.

Here is the create script:

#!/bin/bash

if [ $# != 1 ]; then
exit 1;
fi

if [ -e "$1" ]; then
exit 2;
fi

mkdir "$1"
cd "$1"
git init --bare
cp hooks/post-update.sample hooks/post-update
chmod +x hooks/post-update
git update-server-info




--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.