configuring git public repository

configuring git public repository

[sagi4]

Hi all,

I am new git..

I would like to configure git as a public repository for my rails
application..

Please help me..

Thanks
Sg..
-- 
View this message in context: http://www.nabble.com/configuring-git-public-repository-tp19449377p19449377.html
Sent from the git mailing list archive at Nabble.com.

Re: configuring git public repository

[Ciprian Dorin Craciun]

    Hello!

    If I'm correct, you have two major options:
    -- install your own git infrastructure:
        -- by using git-daemon and serving the git repository throught
git://...;
        -- by using gitweb? and serving it as http://...;
        -- by using ssh and serving it as ssh://... (this is
appropriate mostly for private repositories);
    -- using an existing Git hosting service like:
        -- (of course) http://repo.or.cz/
        -- http://github.org/
        -- http://gitorious.org/
        -- see also http://git.or.cz/gitwiki/GitHosting

    For open-source / public repositories I would opt for using an
existing hosting service.

    Hope it's useful for you,
    Ciprian Dorin Craciun.


On Fri, Sep 12, 2008 at 7:47 AM, sagi4 <geetha@angleritech.com> wrote:
>
> Hi all,
>
> I am new git..
>
> I would like to configure git as a public repository for my rails
> application..
>
> Please help me..
>
> Thanks
> Sg..
> --
> View this message in context: http://www.nabble.com/configuring-git-public-repository-tp19449377p19449377.html
> Sent from the git mailing list archive at Nabble.com.

Re: configuring git public repository

[Jakub Narebski]

"Ciprian Dorin Craciun" <ciprian.craciun@gmail.com> writes:

> On Fri, Sep 12, 2008 at 7:47 AM, sagi4 <geetha@angleritech.com> wrote:
> >
> > Hi all,
> >
> > I am new git..
> >
> > I would like to configure git as a public repository for my rails
> > application..
> >
> > Please help me..

First, I think you can find required information in many documentation
avaliable online, like "Git User's Manual" (distributed with Git), or
Got Community Book from http://git-scm.com
 
> If I'm correct, you have two major options:
> -- install your own git infrastructure:
>    -- by using git-daemon and serving the git repository throught
>       git://...;

Note that git:// protocol is meant only for _fetching_ (although you
can configure it for pushing, it is a good option only in very narrow
set of circumstances). And you would have ensure that your firewall
allows git traffic.

>    -- by using gitweb? and serving it as http://...;

Gitweb (and cgit, and git-php) is _web interface_ to repository, which
means that you can check the state of repository, view current version
and the log of changes, and many other things from a web browser.  All
of those require installing either as CGI module, or some other module
for web server having installed.

To have read-only HTTP access you only need to put your repository in
a place where your web server will see it.  Well, that and ensure that
git-update-server-info is run, for clients to be able to know latest
state of the repository.  Usually it is done using 'update' hook; it
should be enough to enable example hook (if you use bare repository as
published repository).

To be able to push via HTTP you have to set WebDAV and appropriate
permissions on web server. See the documentation.

>    -- by using ssh and serving it as ssh://... (this is
>       appropriate mostly for private repositories);

Usually you would want to restrict access by using git-shell, and
perhaps ease configuring permissions using either Gitosis (see also
BlogPosts on git wiki) or ssh_acl.

>  -- using an existing Git hosting service like:
>     -- (of course) http://repo.or.cz/
>     -- http://github.org/
>     -- http://gitorious.org/
>     -- see also http://git.or.cz/gitwiki/GitHosting
> 
> For open-source / public repositories I would opt for using an
> existing hosting service.

For Rails application I would check what other Ruby on Rails people
use. I would guess that it would be either Rubyforge or GitHub.
 
HTH.
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: configuring git public repository

[Lars Hjemli]

On Fri, Sep 12, 2008 at 2:57 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>
> "Ciprian Dorin Craciun" <ciprian.craciun@gmail.com> writes:
>
> > On Fri, Sep 12, 2008 at 7:47 AM, sagi4 <geetha@angleritech.com> wrote:
> > >
> > > Hi all,
> > >
> > > I am new git..
> > >
> > > I would like to configure git as a public repository for my rails
> > > application..
>
> >    -- by using gitweb? and serving it as http://...;
>
> Gitweb (and cgit, and git-php) is _web interface_ to repository, which
> means that you can check the state of repository, view current version
> and the log of changes, and many other things from a web browser.

<plug>
Current cgit also allows cloning over http using the same url as for
browsing the repo, i.e. you may

  $ git clone http://hjemli.net/git/cgit

This has one advantage over just publishing the repo; you don't have
to run `git-update-server-info` (thanks to the work done by Shawn O.
Pearce on git-http-backend, which is shamelessly reimplemented in
cgit).
</plug>

--
larsh

CGit and repository list

[Petr Baudis]

  Hi,

On Fri, Sep 12, 2008 at 04:36:11PM +0200, Lars Hjemli wrote:
> <plug>
> Current cgit also allows cloning over http using the same url as for
> browsing the repo, i.e. you may
> 
>   $ git clone http://hjemli.net/git/cgit
> 
> This has one advantage over just publishing the repo; you don't have
> to run `git-update-server-info` (thanks to the work done by Shawn O.
> Pearce on git-http-backend, which is shamelessly reimplemented in
> cgit).
> </plug>

  this finally tripped me over and I wanted to quickly add cgit as an
alternate viewing interface at repo.or.cz. But it seems that cgit
requires all the repositories explicitly listed in the config file.
Do you plan to remove this limitation in the future?

  Thanks,

				Petr "Pasky" Baudis

Re: CGit and repository list

[Lars Hjemli]

On Fri, Sep 12, 2008 at 4:58 PM, Petr Baudis <pasky@suse.cz> wrote:
> it seems that cgit
> requires all the repositories explicitly listed in the config file.
> Do you plan to remove this limitation in the future?

Not really, I'd rather add another command (or a commandline option)
to generate an include-file for cgitrc by scanning directory-trees for
git repos. I've CC'd Kristian since I believe he's got such a script
running for freedesktop.org; if so, maybe it could be included/used as
basis for something similar in cgit?

--
larsh

Re: CGit and repository list

[Petr Baudis]

[-- Attachment #1: Type: text/plain, Size: 1981 bytes --]

On Fri, Sep 12, 2008 at 06:12:40PM +0200, Johan Herland wrote:
> On Friday 12 September 2008, Lars Hjemli wrote:
> > On Fri, Sep 12, 2008 at 4:58 PM, Petr Baudis <pasky@suse.cz> wrote:
> > > it seems that cgit
> > > requires all the repositories explicitly listed in the config file.
> > > Do you plan to remove this limitation in the future?
> >
> > Not really, I'd rather add another command (or a commandline option)
> > to generate an include-file for cgitrc by scanning directory-trees
> > for git repos. I've CC'd Kristian since I believe he's got such a
> > script running for freedesktop.org; if so, maybe it could be
> > included/used as basis for something similar in cgit?
> 
> Here's a script I wrote for locating repos and generating repo 
> lists/configs for cgit, gitweb and hgwebdir (yes, this handles hg repos 
> as well). It works either as a CGI script (producing a list of detected 
> repos in HTML format), or from the command-line. It's only been tested 
> on an experimental DVCS server at $dayjob, so you might have to change 
> things to make it work in your scenario.
> 
> If there is interest in this, I can create a public repo and we can keep 
> improving on it.

Thanks. The script was a bit more scary than I thought, but in the end I
managed to generate something. There are trailing dots in project names,
but I'm not going to waste time on that anymore - this has long gone
over the 20 minutes I originally alotted the project anyway; I hope cgit
will gain a builtin capability for this in the future, since this is
still quite a pain. Attached is a random patch for your script I had to
use, FWIW.

Unfortunately, the recommended RewriteRule is not working - it does not
play well together with query parameters cgit is using, so e.g. browsing
past commits does not work. What RewriteRule should I use instead?

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

[-- Attachment #2: repofinder.diff --]
[-- Type: text/plain, Size: 2107 bytes --]

--- repofinder-orig.py	2008-09-13 00:44:22.000000000 +0200
+++ repofinder.py	2008-09-13 00:44:47.000000000 +0200
@@ -244,7 +244,7 @@
 			if (self.group):
 				assert name.startswith(self.group.path)
 				name = name[len(self.group.path):]
-			name = name.strip("/")
+			name = name.strip("/srv/git/")
 			self._name = self.nameFromPath(name)
 		return self._name
 	name = property(getName)
@@ -267,7 +267,10 @@
 		"""Return a (name, email) tuple with our best guess as to who
 		   owns this repo
 		"""
-		return (self.ownerName, self.ownerEmail)
+		__o = self.ownerName
+		if (__o is None):
+			__o = ''
+		return (__o, self.ownerEmail)
 	owner = property(getOwner)
 
 	def getDescription (self):
@@ -286,14 +289,15 @@
 
 	@staticmethod
 	def nameFromPath (path):
-		if path.endswith(".git"): path = path[:-4]
+		if path.endswith(".git"): path = path[:-5]
 		path = path.rstrip("/")
 		return path
 
 	@staticmethod
 	def cloneUrlFromPath (path):
 		if path.endswith("/.git"): path = path[:-5]
-		return "ssh://%s%s" % (Settings['ServerName'], path)
+		path = path.strip("/srv/git")
+		return "git://%s/%sgit" % (Settings['ServerName'], path)
 
 	def __init__ (self, path):
 		Repo.__init__(self, path)
@@ -316,6 +320,9 @@
 	def getOwnerEmail (self):
 		if self._ownerEmail is None:
 			self._ownerEmail = self.config.get("user.email")
+		if self._ownerEmail is None:
+			try: self._ownerEmail = open(os.path.join(self.path, "owner")).read().strip()
+			except: pass
 		return Repo.getOwnerEmail(self)
 	ownerEmail = property(getOwnerEmail)
 
@@ -592,12 +599,12 @@
 		for repo in sorted(group.repos):
 			url = repo.path
 			if url.endswith("/.git"): url = url[:-5]
-			url = url.strip("/")
+			url = url.strip("/srv/git/")
 			print >>self.f, "repo.url=%s" % (url)
 			print >>self.f, "repo.path=%s" % (repo.path)
 			print >>self.f, "repo.name=%s" % (repo.name)
 			print >>self.f, "repo.desc=%s" % (repo.description)
-			print >>self.f, "repo.owner=%s <%s>" % repo.owner
+			print >>self.f, "repo.owner=%s%s" % repo.owner
 			print >>self.f, "repo.clone-url=%s" % (repo.clone)
 			print >>self.f, ""
 

Re: CGit and repository list

[Lars Hjemli]

On Sat, Sep 13, 2008 at 12:48 AM, Petr Baudis <pasky@suse.cz> wrote:
> On Fri, Sep 12, 2008 at 06:12:40PM +0200, Johan Herland wrote:
>> On Friday 12 September 2008, Lars Hjemli wrote:
>> > On Fri, Sep 12, 2008 at 4:58 PM, Petr Baudis <pasky@suse.cz> wrote:
>> > > it seems that cgit
>> > > requires all the repositories explicitly listed in the config file.
>> > > Do you plan to remove this limitation in the future?
>> >
>> > Not really, I'd rather add another command (or a commandline option)
>> > to generate an include-file for cgitrc by scanning directory-trees
>> > for git repos. I've CC'd Kristian since I believe he's got such a
>> > script running for freedesktop.org; if so, maybe it could be
>> > included/used as basis for something similar in cgit?
>>
>> Here's a script I wrote for locating repos and generating repo
>> lists/configs for cgit, gitweb and hgwebdir (yes, this handles hg repos
>> as well). It works either as a CGI script (producing a list of detected
>> repos in HTML format), or from the command-line. It's only been tested
>> on an experimental DVCS server at $dayjob, so you might have to change
>> things to make it work in your scenario.
>>
>> If there is interest in this, I can create a public repo and we can keep
>> improving on it.
>
> Thanks. The script was a bit more scary than I thought, but in the end I
> managed to generate something. There are trailing dots in project names,
> but I'm not going to waste time on that anymore - this has long gone
> over the 20 minutes I originally alotted the project anyway; I hope cgit
> will gain a builtin capability for this in the future, since this is
> still quite a pain.

I guess I could add support for something like

  scan-paths=/pub/git

in cgitrc (and optionally store the result of the scan as another
cgitrc-file in the cache directory). Would that improve things for
you?


> Unfortunately, the recommended RewriteRule is not working - it does not
> play well together with query parameters cgit is using, so e.g. browsing
> past commits does not work. What RewriteRule should I use instead?

On hjemli.net I used to specify "virtual-root=/git" in cgitrc combined
with this rule in /etc/apache/httpd.conf

  RewriteRule ^/git/(.*)$   /cgit/cgit.cgi?url=$1  [L,QSA]


But currently I'm running the tip of the wip-branch (which has support
for PATH_INFO) on hjemli.net , so I've removed the "virtual-root" from
cgitrc and the rewriterule from httpd.conf and just use this instead:

  ScriptAlias /git /var/www/htdocs/cgit/cgit.cgi

hth,
larsh

Re: CGit and repository list

[Petr Baudis]

On Sat, Sep 13, 2008 at 01:20:50AM +0200, Lars Hjemli wrote:
> I guess I could add support for something like
> 
>   scan-paths=/pub/git
> 
> in cgitrc (and optionally store the result of the scan as another
> cgitrc-file in the cache directory). Would that improve things for
> you?

Yes, certainly.

> > Unfortunately, the recommended RewriteRule is not working - it does not
> > play well together with query parameters cgit is using, so e.g. browsing
> > past commits does not work. What RewriteRule should I use instead?
> 
> On hjemli.net I used to specify "virtual-root=/git" in cgitrc combined
> with this rule in /etc/apache/httpd.conf
> 
>   RewriteRule ^/git/(.*)$   /cgit/cgit.cgi?url=$1  [L,QSA]

Ah, I see - sorry. I forgot the [L,QSA] part.

I wonder why

	http://repo.or.cz/c/libc.git/

has such a funny-looking summary page.

Let me know if you want me to update cgit there sometime. In the next
days I will add some means for switching between cgit and gitweb views.

P.S.: Johan - I discovered the cause of my problems - .strip() is
totally inappropriate string method to call here, it takes a _set_ of
characters.

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

Re: CGit and repository list

[Lars Hjemli]

On Sat, Sep 13, 2008 at 9:49 PM, Petr Baudis <pasky@suse.cz> wrote:
> On Sat, Sep 13, 2008 at 01:20:50AM +0200, Lars Hjemli wrote:
>> I guess I could add support for something like
>>
>>   scan-paths=/pub/git
>>
>> in cgitrc (and optionally store the result of the scan as another
>> cgitrc-file in the cache directory). Would that improve things for
>> you?
>
> Yes, certainly.

Ok, I'll try to come up with something.


> I wonder why
>
>        http://repo.or.cz/c/libc.git/
>
> has such a funny-looking summary page.

I suppose you mean http://repo.or.cz/c/glibc.git/? It looks like cgit
dies when trying to process the tags in this repo. I'll clone it and
see if I can reproduce/fix the problem.


> Let me know if you want me to update cgit there sometime. In the next
> days I will add some means for switching between cgit and gitweb views.

Cool, I'll keep you posted ;-)

--
larsh

[CGIT PATCH] parsing.c: handle unexpected commit/tag content

[Lars Hjemli]

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---

Can you test this on top of the current master? It seems to fix the problems
with glibc.git.


 cgit.h    |    2 +-
 parsing.c |  108 +++++++++++++++++++++++++++++++++++++++---------------------
 2 files changed, 71 insertions(+), 39 deletions(-)

diff --git a/cgit.h b/cgit.h
index 1615616..08fd95a 100644
--- a/cgit.h
+++ b/cgit.h
@@ -85,7 +85,7 @@ struct commitinfo {
 struct taginfo {
 	char *tagger;
 	char *tagger_email;
-	int tagger_date;
+	unsigned long tagger_date;
 	char *msg;
 };

diff --git a/parsing.c b/parsing.c
index 66e8b3d..0396133 100644
--- a/parsing.c
+++ b/parsing.c
@@ -62,6 +62,40 @@ char *substr(const char *head, const char *tail)
 	return buf;
 }

+char *parse_user(char *t, char **name, char **email, unsigned long *date)
+{
+	char *p = t;
+	int mode = 1;
+
+	while (p && *p) {
+		if (mode == 1 && *p == '<') {
+			*name = substr(t, p - 1);
+			t = p;
+			mode++;
+		} else if (mode == 1 && *p == '\n') {
+			*name = substr(t, p);
+			p++;
+			break;
+		} else if (mode == 2 && *p == '>') {
+			*email = substr(t, p + 1);
+			t = p;
+			mode++;
+		} else if (mode == 2 && *p == '\n') {
+			*email = substr(t, p);
+			p++;
+			break;
+		} else if (mode == 3 && isdigit(*p)) {
+			*date = atol(p);
+			mode++;
+		} else if (*p == '\n') {
+			p++;
+			break;
+		}
+		p++;
+	}
+	return p;
+}
+
 struct commitinfo *cgit_parse_commit(struct commit *commit)
 {
 	struct commitinfo *ret;
@@ -88,29 +122,17 @@ struct commitinfo *cgit_parse_commit(struct commit *commit)
 	while (!strncmp(p, "parent ", 7))
 		p += 48; // "parent " + hex[40] + "\n"

-	if (!strncmp(p, "author ", 7)) {
-		p += 7;
-		t = strchr(p, '<') - 1;
-		ret->author = substr(p, t);
-		p = t;
-		t = strchr(t, '>') + 1;
-		ret->author_email = substr(p, t);
-		ret->author_date = atol(t+1);
-		p = strchr(t, '\n') + 1;
+	if (p && !strncmp(p, "author ", 7)) {
+		p = parse_user(p + 7, &ret->author, &ret->author_email,
+			&ret->author_date);
 	}

-	if (!strncmp(p, "committer ", 9)) {
-		p += 9;
-		t = strchr(p, '<') - 1;
-		ret->committer = substr(p, t);
-		p = t;
-		t = strchr(t, '>') + 1;
-		ret->committer_email = substr(p, t);
-		ret->committer_date = atol(t+1);
-		p = strchr(t, '\n') + 1;
+	if (p && !strncmp(p, "committer ", 9)) {
+		p = parse_user(p + 9, &ret->committer, &ret->committer_email,
+			&ret->committer_date);
 	}

-	if (!strncmp(p, "encoding ", 9)) {
+	if (p && !strncmp(p, "encoding ", 9)) {
 		p += 9;
 		t = strchr(p, '\n') + 1;
 		ret->msg_encoding = substr(p, t);
@@ -118,25 +140,38 @@ struct commitinfo *cgit_parse_commit(struct commit *commit)
 	} else
 		ret->msg_encoding = xstrdup(PAGE_ENCODING);

-	while (*p && (*p != '\n'))
-		p = strchr(p, '\n') + 1; // skip unknown header fields
+	// skip unknown header fields
+	while (p && *p && (*p != '\n')) {
+		p = strchr(p, '\n');
+		if (p)
+			p++;
+	}

-	while (*p == '\n')
-		p = strchr(p, '\n') + 1;
+	// skip extra blank lines between headers and message body
+	while (p && *p == '\n') {
+		p = strchr(p, '\n');
+		if (p)
+			p++;
+	}

 	t = strchr(p, '\n');
 	if (t) {
-		if (*t == '\0')
+		if (*t == '\0') {
 			ret->subject = "** empty **";
-		else
+			return ret;
+		} else {
 			ret->subject = substr(p, t);
-		p = t + 1;
+			p = t + 1;
+		}

-		while (*p == '\n')
-			p = strchr(p, '\n') + 1;
+		while (p && *p == '\n') {
+			p = strchr(p, '\n');
+			if (p)
+				p++;
+		}
 		ret->msg = xstrdup(p);
 	} else
-		ret->subject = substr(p, p+strlen(p));
+		ret->subject = xstrdup(p);

 	if(strcmp(ret->msg_encoding, PAGE_ENCODING)) {
 		t = reencode_string(ret->subject, PAGE_ENCODING,
@@ -163,7 +198,7 @@ struct taginfo *cgit_parse_tag(struct tag *tag)
 	void *data;
 	enum object_type type;
 	unsigned long size;
-	char *p, *t;
+	char *p;
 	struct taginfo *ret;

 	data = read_sha1_file(tag->object.sha1, &type, &size);
@@ -185,15 +220,12 @@ struct taginfo *cgit_parse_tag(struct tag *tag)
 			break;

 		if (!strncmp(p, "tagger ", 7)) {
-			p += 7;
-			t = strchr(p, '<') - 1;
-			ret->tagger = substr(p, t);
-			p = t;
-			t = strchr(t, '>') + 1;
-			ret->tagger_email = substr(p, t);
-			ret->tagger_date = atol(t+1);
+			p = parse_user(p + 7, &ret->tagger, &ret->tagger_email,
+				&ret->tagger_date);
 		}
-		p = strchr(p, '\n') + 1;
+		p = strchr(p, '\n');
+		if (p)
+			p++;
 	}

 	while (p && *p && (*p != '\n'))
--
1.6.0.rc1

Re: [CGIT PATCH] parsing.c: handle unexpected commit/tag content

[Lars Hjemli]

On Sun, Sep 14, 2008 at 9:53 AM, Lars Hjemli <hjemli@gmail.com> wrote:
> +       // skip extra blank lines between headers and message body
> +       while (p && *p == '\n') {
> +               p = strchr(p, '\n');
> +               if (p)
> +                       p++;
> +       }

Btw, this is just silly so I've changed it:

while (p && *p == '\n')
        p++;

The wip-branch at http://hjemli.net/git/cgit has a fixed-up patch.

--
larsh

Re: CGit and repository list

[Lars Hjemli]

On Sat, Sep 13, 2008 at 9:49 PM, Petr Baudis <pasky@suse.cz> wrote:
> On Sat, Sep 13, 2008 at 01:20:50AM +0200, Lars Hjemli wrote:
>> I guess I could add support for something like
>>
>>   scan-paths=/pub/git
>>
>> in cgitrc (and optionally store the result of the scan as another
>> cgitrc-file in the cache directory). Would that improve things for
>> you?
>
> Yes, certainly.

Ok, for now I've added a --scan-tree=<path> commandline option which
will make cgit print a sorted list of the repositories found to
stdout. If/when this does a reasonable job on repo.or.cz I can look
into some form of automatic rescanning (enabled from cgitrc).


>> > Unfortunately, the recommended RewriteRule is not working - it does not
>> > play well together with query parameters cgit is using, so e.g. browsing
>> > past commits does not work.

I've also pushed support for PATH_INFO. This allows you to drop the
rewriterules from httpd.conf and the 'virtual-url' option from cgitrc
and still get the 'nice' urls.


> I wonder why
>
>        http://repo.or.cz/c/libc.git/
>
> has such a funny-looking summary page.

It's due to cgits terrible functions for parsing commits and tags.
I've pushed a reworked patch which should make the parsing more robust
and thus also render glibc.git correctly.

Finally, and probably more interesting for freedesktop.org than
repo.or.cz, is the support for repolist filtering by url prefix
(compare http://hjemli.net/git and http://hjemli.net/git/xorg for a
minimal example).

This is all available from the master branch in
git://hjemli.net/pub/git/cgit, enjoy!

--
larsh

Re: CGit and repository list

[Kristian Høgsberg]

On Fri, Sep 12, 2008 at 6:48 PM, Petr Baudis <pasky@suse.cz> wrote:
> On Fri, Sep 12, 2008 at 06:12:40PM +0200, Johan Herland wrote:
>> On Friday 12 September 2008, Lars Hjemli wrote:
>> > On Fri, Sep 12, 2008 at 4:58 PM, Petr Baudis <pasky@suse.cz> wrote:
>> > > it seems that cgit
>> > > requires all the repositories explicitly listed in the config file.
>> > > Do you plan to remove this limitation in the future?
>> >
>> > Not really, I'd rather add another command (or a commandline option)
>> > to generate an include-file for cgitrc by scanning directory-trees
>> > for git repos. I've CC'd Kristian since I believe he's got such a
>> > script running for freedesktop.org; if so, maybe it could be
>> > included/used as basis for something similar in cgit?
>>
>> Here's a script I wrote for locating repos and generating repo
>> lists/configs for cgit, gitweb and hgwebdir (yes, this handles hg repos
>> as well). It works either as a CGI script (producing a list of detected
>> repos in HTML format), or from the command-line. It's only been tested
>> on an experimental DVCS server at $dayjob, so you might have to change
>> things to make it work in your scenario.
>>
>> If there is interest in this, I can create a public repo and we can keep
>> improving on it.
>
> Thanks. The script was a bit more scary than I thought, but in the end I
> managed to generate something. There are trailing dots in project names,
> but I'm not going to waste time on that anymore - this has long gone
> over the 20 minutes I originally alotted the project anyway; I hope cgit
> will gain a builtin capability for this in the future, since this is
> still quite a pain. Attached is a random patch for your script I had to
> use, FWIW.

The script we're using on freedesktop.org is something like this:

  http://people.freedesktop.org/~krh/find-repos.sh

which isn't too scary.  We run it from a cron script, every ten
minutes or so.  It's much better than  rescanning all the dirs every
time somebody requests the repo listing (we have a lot of repos, check
cgit.freedesktop.org).  We have no need for cgit to to this
automatically on every request, and I don't see us using that feature
if it gets implemented.

cheers,
Kristian

Re: CGit and repository list

[Kristian Høgsberg]

On Sun, Sep 14, 2008 at 1:14 PM, Kristian Høgsberg <krh@bitplanet.net> wrote:
> The script we're using on freedesktop.org is something like this:
>
>  http://people.freedesktop.org/~krh/find-repos.sh
>
> which isn't too scary.  We run it from a cron script, every ten
> minutes or so.  It's much better than  rescanning all the dirs every
> time somebody requests the repo listing (we have a lot of repos, check
> cgit.freedesktop.org).  We have no need for cgit to to this
> automatically on every request, and I don't see us using that feature
> if it gets implemented.

A feature that would be useful, though, would be support for listing a
sub-set of repos... for example, cgit.freedesktop.org/~krh would list
just my repos and cgit.freedesktop.org/xorg/driver would list all the
X.org driver repos.  Another idea is a javascript/css frontpage that
supports folding repos away according to the groups and possibly
common path elements in the url.  For example:

  [+] Freedesktop Repositories
  [-] User repositories
      [+] ~agdf5
      [+] ~ahuillet
      ...
      [-] ~krh
           ~krh/akamaru
           ~krh/bugzilla
           ~krh/compiz
      [-] ~kyle

etc.

Just throwing out a couple of ideas.

cheers,
Kristian

Re: CGit and repository list

[Lars Hjemli]

On Sun, Sep 14, 2008 at 7:22 PM, Kristian Høgsberg <krh@bitplanet.net> wrote:
> On Sun, Sep 14, 2008 at 1:14 PM, Kristian Høgsberg <krh@bitplanet.net> wrote:
>> The script we're using on freedesktop.org is something like this:
>>
>>  http://people.freedesktop.org/~krh/find-repos.sh

Thanks.

>>
>> which isn't too scary.  We run it from a cron script, every ten
>> minutes or so.  It's much better than  rescanning all the dirs every
>> time somebody requests the repo listing (we have a lot of repos, check
>> cgit.freedesktop.org).  We have no need for cgit to to this
>> automatically on every request, and I don't see us using that feature
>> if it gets implemented.

Yeah, I'm not interested in a full rescan for each request either. But
having something similar to your script available as a command line
option for cgit wouldn't hurt.


> A feature that would be useful, though, would be support for listing a
> sub-set of repos... for example, cgit.freedesktop.org/~krh would list
> just my repos and cgit.freedesktop.org/xorg/driver would list all the
> X.org driver repos.

This sounds nice, and shouldn't be very difficult...


>  Another idea is a javascript/css frontpage that
> supports folding repos away according to the groups and possibly
> common path elements in the url.

This could also be nice as long as non-javascript clients doesn't suffer ;-)

--
larsh

Re: CGit and repository list

[Jakub Narebski]

Petr Baudis wrote:
> On Fri, Sep 12, 2008 at 04:36:11PM +0200, Lars Hjemli wrote:
> >
> > <plug>
> > Current cgit also allows cloning over http using the same url as for
> > browsing the repo, i.e. you may
> > 
> >   $ git clone http://hjemli.net/git/cgit
> > 
> > This has one advantage over just publishing the repo; you don't have
> > to run `git-update-server-info` (thanks to the work done by Shawn O.
> > Pearce on git-http-backend, which is shamelessly reimplemented in
> > cgit).
> > </plug>
> 
>   this finally tripped me over and I wanted to quickly add cgit as an
> alternate viewing interface at repo.or.cz. [...]

Or you can wait a little while for "smart" HTTP server, which I guess
also generates automatically or/and on the fly objects/info/packs and
info/refs required by "dumb" protocols clients (including old HTTP
clients).

P.S. Could you please gather some statistics to compare the period
before and after installing "smart" HTTP server (and after smart
clients became widespread).

-- 
Jakub Narebski
Poland

Re: CGit and repository list

[Shawn O. Pearce]

Jakub Narebski <jnareb@gmail.com> wrote:
> Petr Baudis wrote:
> > 
> >   this finally tripped me over and I wanted to quickly add cgit as an
> > alternate viewing interface at repo.or.cz. [...]
> 
> Or you can wait a little while for "smart" HTTP server, which I guess
> also generates automatically or/and on the fly objects/info/packs and
> info/refs required by "dumb" protocols clients (including old HTTP
> clients).

The automatic generation of objects/info/packs and info/refs is
planned for support in the smart CGI, but it doesn't help the
"gitweb URL is same as clone URL" concept.  For that you need your
HTTP server to know how to issue some requests to gitweb and others
to the smart CGI or to the filesystem.
 
> P.S. Could you please gather some statistics to compare the period
> before and after installing "smart" HTTP server (and after smart
> clients became widespread).

Well, it would help if there was a working implementation of the
"smart" HTTP server.  ;-)

Right now I haven't been able to touch the project for two weeks
straight.  I'm hoping to be able to put 2 full days into it next
week.  Lets see how that plan works.

-- 
Shawn.

Re: CGit and repository list

[Lars Hjemli]

On Fri, Sep 12, 2008 at 6:00 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:
>> Petr Baudis wrote:
>> >
>> >   this finally tripped me over and I wanted to quickly add cgit as an
>> > alternate viewing interface at repo.or.cz. [...]
>>
>> Or you can wait a little while for "smart" HTTP server, which I guess
>> also generates automatically or/and on the fly objects/info/packs and
>> info/refs required by "dumb" protocols clients (including old HTTP
>> clients).
>
> The automatic generation of objects/info/packs and info/refs is
> planned for support in the smart CGI, but it doesn't help the
> "gitweb URL is same as clone URL" concept.  For that you need your
> HTTP server to know how to issue some requests to gitweb and others
> to the smart CGI or to the filesystem.

In the case of cgit this should work transparently.

>
>> P.S. Could you please gather some statistics to compare the period
>> before and after installing "smart" HTTP server (and after smart
>> clients became widespread).
>
> Well, it would help if there was a working implementation of the
> "smart" HTTP server.  ;-)

I'll try to come up with a prototype for the smart server as part of cgit ;-)

--
larsh

Re: CGit and repository list

[Petr Baudis]

On Fri, Sep 12, 2008 at 05:54:29PM +0200, Jakub Narebski wrote:
> Petr Baudis wrote:
> > On Fri, Sep 12, 2008 at 04:36:11PM +0200, Lars Hjemli wrote:
> > >
> > > <plug>
> > > Current cgit also allows cloning over http using the same url as for
> > > browsing the repo, i.e. you may
> > > 
> > >   $ git clone http://hjemli.net/git/cgit
> > > 
> > > This has one advantage over just publishing the repo; you don't have
> > > to run `git-update-server-info` (thanks to the work done by Shawn O.
> > > Pearce on git-http-backend, which is shamelessly reimplemented in
> > > cgit).
> > > </plug>
> > 
> >   this finally tripped me over and I wanted to quickly add cgit as an
> > alternate viewing interface at repo.or.cz. [...]
> 
> Or you can wait a little while for "smart" HTTP server, which I guess
> also generates automatically or/and on the fly objects/info/packs and
> info/refs required by "dumb" protocols clients (including old HTTP
> clients).

It is more like that this prodded me to have a look at cgit again, think
of it and realize that it should be pretty easy for me to add cgit to
repo.or.cz. This is just a bonus. :-)

I have already mailed Shawn immediately after the original proposal
was posted that I'm willing to put this up on repo.or.cz as soon as
something tangible is ready.

> P.S. Could you please gather some statistics to compare the period
> before and after installing "smart" HTTP server (and after smart
> clients became widespread).

What kind of statistics?

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

Re: CGit and repository list

[Shawn O. Pearce]

Petr Baudis <pasky@suse.cz> wrote:
> On Fri, Sep 12, 2008 at 05:54:29PM +0200, Jakub Narebski wrote:
> 
> > P.S. Could you please gather some statistics to compare the period
> > before and after installing "smart" HTTP server (and after smart
> > clients became widespread).
> 
> What kind of statistics?

Disk IO and network IO consumed probably.  The kernel.org folks are
hoping the smart HTTP server can lower their loads a bit by being
more careful about what we send to the client.

-- 
Shawn.

Re: CGit and repository list

[Petr Baudis]

On Fri, Sep 12, 2008 at 09:08:54AM -0700, Shawn O. Pearce wrote:
> Petr Baudis <pasky@suse.cz> wrote:
> > On Fri, Sep 12, 2008 at 05:54:29PM +0200, Jakub Narebski wrote:
> > 
> > > P.S. Could you please gather some statistics to compare the period
> > > before and after installing "smart" HTTP server (and after smart
> > > clients became widespread).
> > 
> > What kind of statistics?
> 
> Disk IO and network IO consumed probably.  The kernel.org folks are
> hoping the smart HTTP server can lower their loads a bit by being
> more careful about what we send to the client.

To check whether it actually matters for me, I have counted HTTP
requests for info/refs: 42 per hour for the last 5.5 days. So it might.
28% of the requests are web crawlers.

For objects/../, it is more fun - 1942 requests per hour. 46% is
accounted for web crawlers. I will put up a robots.txt. ;-)

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

Re: CGit and repository list

[Shawn O. Pearce]

Petr Baudis <pasky@suse.cz> wrote:
> > 
> > Disk IO and network IO consumed probably.  The kernel.org folks are
> > hoping the smart HTTP server can lower their loads a bit by being
> > more careful about what we send to the client.
> 
> To check whether it actually matters for me, I have counted HTTP
> requests for info/refs: 42 per hour for the last 5.5 days. So it might.
> 28% of the requests are web crawlers.
> 
> For objects/../, it is more fun - 1942 requests per hour. 46% is
> accounted for web crawlers. I will put up a robots.txt. ;-)

Is Googlebot trying to make those loose objects searchable?
I wonder what byte sequence I should try in the search bar...

;-)

-- 
Shawn.

Re: CGit and repository list

[Jakub Narebski]

Dnia piątek 12. września 2008 18:05, Petr Baudis napisał:
> On Fri, Sep 12, 2008 at 05:54:29PM +0200, Jakub Narebski wrote:
>>
>> P.S. Could you please gather some statistics to compare the period
>> before and after installing "smart" HTTP server (and after smart
>> clients became widespread).
> 
> What kind of statistics?

In the simplest case just some averaged system load (perhaps load 
average from uptime, or from top, or from /proc) before and after.
Perhaps bandwidth used per week or something like that too.

If you have time and interest, CPU load, disk IO, network IO or
network bandwidth used, and average time to serve request (latency)
for fetching via HTTP protocol, for "dumb" and "smart" clients,
perhaps averaged over number of requests.  (I don't know unfortunately 
how to get such data).

In short: check how "smart" HTTP protocol would improve things.

-- 
Jakub Narebski
Poland

Re: CGit and repository list

[Shawn O. Pearce]

Jakub Narebski <jnareb@gmail.com> wrote:
> Dnia piątek 12. września 2008 18:05, Petr Baudis napisał:
> > On Fri, Sep 12, 2008 at 05:54:29PM +0200, Jakub Narebski wrote:
> >>
> >> P.S. Could you please gather some statistics to compare the period
> >> before and after installing "smart" HTTP server (and after smart
> >> clients became widespread).
> > 
> > What kind of statistics?
> 
> In short: check how "smart" HTTP protocol would improve things.

The real improvement is on the client side; how long does it take
to do a clone or a fetch with dumb HTTP vs. with smart HTTP.  The
latency is horrible for dumb HTTP if the repository is not packed,
or bad if it just repacked and you have to download everything again.

Knowing the transfer time of each request start-to-finish and the
total time for a "session" (e.g. a period of where that IP appears
actively requesting in the logs) would tell you how long a given
client needs to do a fetch or clone.

-- 
Shawn.

Re: CGit and repository list

[Petr Baudis]

On Fri, Sep 12, 2008 at 07:40:43PM +0200, Jakub Narebski wrote:
> Dnia piątek 12. września 2008 18:05, Petr Baudis napisał:
> > On Fri, Sep 12, 2008 at 05:54:29PM +0200, Jakub Narebski wrote:
> >>
> >> P.S. Could you please gather some statistics to compare the period
> >> before and after installing "smart" HTTP server (and after smart
> >> clients became widespread).
> > 
> > What kind of statistics?
> 
> In the simplest case just some averaged system load (perhaps load 
> average from uptime, or from top, or from /proc) before and after.
> Perhaps bandwidth used per week or something like that too.

This statistics are already available at

	http://rover.dkm.cz/~yanek/stats/

> If you have time and interest, CPU load, disk IO, network IO or
> network bandwidth used, and average time to serve request (latency)
> for fetching via HTTP protocol, for "dumb" and "smart" clients,
> perhaps averaged over number of requests.  (I don't know unfortunately 
> how to get such data).
> 
> In short: check how "smart" HTTP protocol would improve things.

...and I don't know out of the top of my head how to best gather the
rest either. ;-) (If someone is really interested about all this, that's
fine, but they have to give me a pre-made cookbook of recipes how to
hook that up to RRD.)

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates