repo.or.cz renovated

[Petr Baudis <pasky@suse.cz>]

On Sat, Mar 15, 2008 at 02:44:42PM -0700, Jakub Narebski wrote:
> Petr Baudis <pasky@suse.cz> writes:
> 
> > On repo.or.cz (permanently I/O overloaded and hosting 1050 project +
> > forks), 
> 
> It looks like repo.or.cz is overwhelmed by its success. I hope that
> now that there are other software hosting sites with git hosting
> (Savannah, GitHub, Gitorious,...) the number of projects wouldn't grow
> as rapidly.

Actually, it was overwhelmed to so much by its success but by lack of
good maintenance. ;-) I gave it some love again for the past week and
the improvement was, well, overwhelming. :-)

I finally fixed tons of failures and broken repositories, and most
importantly repacked some of the big repositories with object databases
in pretty horrid shape. The effect has been immense, having everything
in database of 1/3 the size and single big pack drastically reduced the
I/O load.

Scenario: Site with about 1100 repositories weighting 13GB, running a
fetch job for about 200 of them hourly. About two git-daemon requests
per minute and 10 gitweb requests per minute (the last two numbers are
taken quite sloppily over a small sample of the last ten minutes ;-).
Site is running on 2x1GHz P3 with 2G RAM, repository is on hw RAID5.
(We are currently preparing to migrate it to a more powerful machine.)

Before, the load on the server would be normally about 6 to 15 _all the
time_ and bunch of git-related processes would be permanently eating
some CPU and crunch on the disk.

After introducing the index caching and repacking the repositories, the
load seems to be around 1 at most and hardly seems to come above 3; all
feels very snappy.

So for anyone running a hosting site, make sure your repositories are
nicely packed. It makes huge difference to the I/O load!

> Another solution would be to divide projects list page into pages,
> perhaps adding search box for searching for a project (by name, by
> description and by owner).
> 
> Nevertheless even with pagination, if we want to have "sort by last
> update" we do need caching.

Yes, I'm pondering about pagination, but because of web clients, not the
server load; it takes firefox on my notebook noticeable time to render
this list already, and it's rather big too. Ideas are welcome here.

My current plan is to have a [Search project] box at the front page,
together with direct link to 'show all'. Other than that, what makes
sense to display on the front page? I think recently added projects (age
< 1 week) for sure. I'm not so sure about recently changed projects -
maybe it is better to keep the front page cruft-free.

-- 
				Petr "Pasky" Baudis
Whatever you can do, or dream you can, begin it.
Boldness has genius, power, and magic in it.	-- J. W. von Goethe