[Qemu-devel] https:// for GIT submodules / "smart" HTTP protocol on git.qemu.git

[Qemu-devel] https:// for GIT submodules / "smart" HTTP protocol on git.qemu.git

[Daniel P. Berrangé]

Most of QEMU regular developers are fortunate to have unfiltered internet
access which allows using the git:// protocol. People working in more
old fashioned / paranoid environments often find access to anything other
than http(s) blocked by firewalls. They'll get a bad experiance when trying
to build QEMU, when we try to initialize the git submodules and git://
protocol fails to connect.

We could improve life for people behind such nasty firewalls if we made our
GIT submodules use https:// URIs, instead of git:// URIs. There is one snag.

Historically git:// URIs were strongly preferred over http(s):// URIs for
reasons for performance. Modern GIT though, supports both the traditional
"dumb" HTTP protocol (which is horribly slow) and a modern "smart" HTTP
protocol whose performance matches that of git:/// URIs.

Unfortunately it seems that git.qemu.org is not configured to enable the
smart HTTP protocol.

So can we get a todo item for git.qemu.org get the web server setup to
enable the smart GIT protocol, so we can then enable use of http:// by
default for all the submodules to improve life for contributors.

FYI, you can validate if a server supports the smart protocol or not
by querying  $REPO_URI/info/refs?service=git-upload-pack and seeing
if the response includes a header

   Content-Type: application/x-git-upload-pack-advertisement

qemu.org does not:

  $ wget -S  'https://git.qemu.org/git/qemu.git/info/refs?service=git-upload-pack' 2>&1 | grep Content-Type

and shows terrible git clone performance for http:// uris

but github for example does

  $ wget -S  'https://github.com/qemu/qemu.git/info/refs?service=git-upload-pack' 2>&1 | grep Content-Type
  Content-Type: application/x-git-upload-pack-advertisement

and shows great performance for http:// URIs

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] https:// for GIT submodules / "smart" HTTP protocol on git.qemu.git

[Daniel P. Berrangé]

On Mon, Mar 19, 2018 at 04:49:39PM +0000, Daniel P. Berrangé wrote:
> Most of QEMU regular developers are fortunate to have unfiltered internet
> access which allows using the git:// protocol. People working in more
> old fashioned / paranoid environments often find access to anything other
> than http(s) blocked by firewalls. They'll get a bad experiance when trying
> to build QEMU, when we try to initialize the git submodules and git://
> protocol fails to connect.
> 
> We could improve life for people behind such nasty firewalls if we made our
> GIT submodules use https:// URIs, instead of git:// URIs. There is one snag.
> 
> Historically git:// URIs were strongly preferred over http(s):// URIs for
> reasons for performance. Modern GIT though, supports both the traditional
> "dumb" HTTP protocol (which is horribly slow) and a modern "smart" HTTP
> protocol whose performance matches that of git:/// URIs.
> 
> Unfortunately it seems that git.qemu.org is not configured to enable the
> smart HTTP protocol.
> 
> So can we get a todo item for git.qemu.org get the web server setup to
> enable the smart GIT protocol, so we can then enable use of http:// by
> default for all the submodules to improve life for contributors.

FYI this doesn't seem to take much more than adding this to Apache
config:

  SetEnv GIT_PROJECT_ROOT /path/to/git/projects/root

  <Directory /usr/libexec/git-core>
   Require all granted
  </Directory>

  ScriptAliasMatch "^/git/(.*\.git/(HEAD|info/refs))$" \
    /usr/libexec/git-core/git-http-backend/$1
  ScriptAliasMatch "^/git/(.*\.git/git-(upload|receive)-pack)$" \
    /usr/libexec/git-core/git-http-backend/$1


And then touching the file 'git-daemon-export-ok' in each git repo
that is intended to be public.

  https://git.seveas.net/apache-gitweb-cgit-smart-http.html

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] https:// for GIT submodules / "smart" HTTP protocol on git.qemu.git

[Jeff Cody]

On Mon, Mar 19, 2018 at 06:28:16PM +0000, Daniel P. Berrangé wrote:
> On Mon, Mar 19, 2018 at 04:49:39PM +0000, Daniel P. Berrangé wrote:
> > Most of QEMU regular developers are fortunate to have unfiltered internet
> > access which allows using the git:// protocol. People working in more
> > old fashioned / paranoid environments often find access to anything other
> > than http(s) blocked by firewalls. They'll get a bad experiance when trying
> > to build QEMU, when we try to initialize the git submodules and git://
> > protocol fails to connect.
> > 
> > We could improve life for people behind such nasty firewalls if we made our
> > GIT submodules use https:// URIs, instead of git:// URIs. There is one snag.
> > 
> > Historically git:// URIs were strongly preferred over http(s):// URIs for
> > reasons for performance. Modern GIT though, supports both the traditional
> > "dumb" HTTP protocol (which is horribly slow) and a modern "smart" HTTP
> > protocol whose performance matches that of git:/// URIs.
> > 
> > Unfortunately it seems that git.qemu.org is not configured to enable the
> > smart HTTP protocol.
> > 
> > So can we get a todo item for git.qemu.org get the web server setup to
> > enable the smart GIT protocol, so we can then enable use of http:// by
> > default for all the submodules to improve life for contributors.
> 
> FYI this doesn't seem to take much more than adding this to Apache
> config:
> 
>   SetEnv GIT_PROJECT_ROOT /path/to/git/projects/root
> 
>   <Directory /usr/libexec/git-core>
>    Require all granted
>   </Directory>
> 
>   ScriptAliasMatch "^/git/(.*\.git/(HEAD|info/refs))$" \
>     /usr/libexec/git-core/git-http-backend/$1
>   ScriptAliasMatch "^/git/(.*\.git/git-(upload|receive)-pack)$" \
>     /usr/libexec/git-core/git-http-backend/$1
> 
> 
> And then touching the file 'git-daemon-export-ok' in each git repo
> that is intended to be public.
> 
>   https://git.seveas.net/apache-gitweb-cgit-smart-http.html
> 

Daniel,

This is a good idea.  I'll look into implementing it (I'll wait until a time
of day that is relatively low-traffic).

-Jeff