* [PATCH] gitweb: speed up project listing by limiting find depth
@ 2007-10-17 1:03 Luke Lu
0 siblings, 0 replies; 6+ messages in thread
From: Luke Lu @ 2007-10-17 1:03 UTC (permalink / raw)
To: git; +Cc: pasky, Luke Lu
Resubmit patch based on feedback from Shawn O. Pearce.
Signed-off-by: Luke Lu <git@vicaya.com>
---
Makefile | 2 ++
gitweb/gitweb.perl | 11 +++++++++++
2 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/Makefile b/Makefile
index 8db4dbe..3e9938e 100644
--- a/Makefile
+++ b/Makefile
@@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl
GITWEB_HOME_LINK_STR = projects
GITWEB_SITENAME =
GITWEB_PROJECTROOT = /pub/git
+GITWEB_PROJECT_MAXDEPTH = 2007
GITWEB_EXPORT_OK =
GITWEB_STRICT_EXPORT =
GITWEB_BASE_URL =
@@ -831,6 +832,7 @@ gitweb/gitweb.cgi: gitweb/gitweb.perl
-e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
-e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
-e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
+ -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
-e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
-e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
-e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 3064298..5eb4414 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -35,6 +35,10 @@ our $GIT = "++GIT_BINDIR++/git";
#our $projectroot = "/pub/scm";
our $projectroot = "++GITWEB_PROJECTROOT++";
+# fs traversing limit for getting project list
+# the number is relative to the projectroot
+our $project_maxdepth = "++GITWEB_PROJECT_MAXDEPTH++";
+
# target of the home link on top of all pages
our $home_link = $my_uri || "/";
@@ -1509,16 +1513,23 @@ sub git_get_projects_list {
# remove the trailing "/"
$dir =~ s!/+$!!;
my $pfxlen = length("$dir");
+ my $pfxdepth = ($dir =~ tr!/!!);
File::Find::find({
follow_fast => 1, # follow symbolic links
follow_skip => 2, # ignore duplicates
+ no_chdir => 1, # don't chdir into every directory
dangling_symlinks => 0, # ignore dangling symlinks, silently
wanted => sub {
# skip project-list toplevel, if we get it.
return if (m!^[/.]$!);
# only directories can be git repositories
return unless (-d $_);
+ # don't traverse too deep (Find is super slow on os x)
+ if (tr!/!! - $pfxdepth > $project_maxdepth) {
+ $File::Find::prune = 1;
+ return;
+ }
my $subdir = substr($File::Find::name, $pfxlen + 1);
# we check related file in $projectroot
--
1.5.3.4
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH] gitweb: speed up project listing by limiting find depth
@ 2007-10-17 1:13 Luke Lu
2007-10-17 1:35 ` Shawn O. Pearce
2007-10-17 2:40 ` Shawn O. Pearce
0 siblings, 2 replies; 6+ messages in thread
From: Luke Lu @ 2007-10-17 1:13 UTC (permalink / raw)
To: git; +Cc: pasky, Luke Lu
Resubmit patch due to tab/space issue :)
Signed-off-by: Luke Lu <git@vicaya.com>
---
Makefile | 2 ++
gitweb/gitweb.perl | 11 +++++++++++
2 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/Makefile b/Makefile
index 8db4dbe..3e9938e 100644
--- a/Makefile
+++ b/Makefile
@@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl
GITWEB_HOME_LINK_STR = projects
GITWEB_SITENAME =
GITWEB_PROJECTROOT = /pub/git
+GITWEB_PROJECT_MAXDEPTH = 2007
GITWEB_EXPORT_OK =
GITWEB_STRICT_EXPORT =
GITWEB_BASE_URL =
@@ -831,6 +832,7 @@ gitweb/gitweb.cgi: gitweb/gitweb.perl
-e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
-e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
-e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
+ -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
-e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
-e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
-e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 3064298..d62357f 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -35,6 +35,10 @@ our $GIT = "++GIT_BINDIR++/git";
#our $projectroot = "/pub/scm";
our $projectroot = "++GITWEB_PROJECTROOT++";
+# fs traversing limit for getting project list
+# the number is relative to the projectroot
+our $project_maxdepth = "++GITWEB_PROJECT_MAXDEPTH++";
+
# target of the home link on top of all pages
our $home_link = $my_uri || "/";
@@ -1509,16 +1513,23 @@ sub git_get_projects_list {
# remove the trailing "/"
$dir =~ s!/+$!!;
my $pfxlen = length("$dir");
+ my $pfxdepth = ($dir =~ tr!/!!);
File::Find::find({
follow_fast => 1, # follow symbolic links
follow_skip => 2, # ignore duplicates
+ no_chdir => 1, # don't chdir into every directory
dangling_symlinks => 0, # ignore dangling symlinks, silently
wanted => sub {
# skip project-list toplevel, if we get it.
return if (m!^[/.]$!);
# only directories can be git repositories
return unless (-d $_);
+ # don't traverse too deep (Find is super slow on os x)
+ if (tr!/!! - $pfxdepth > $project_maxdepth) {
+ $File::Find::prune = 1;
+ return;
+ }
my $subdir = substr($File::Find::name, $pfxlen + 1);
# we check related file in $projectroot
--
1.5.3.4
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] gitweb: speed up project listing by limiting find depth
2007-10-17 1:13 Luke Lu
@ 2007-10-17 1:35 ` Shawn O. Pearce
2007-10-17 2:40 ` Shawn O. Pearce
1 sibling, 0 replies; 6+ messages in thread
From: Shawn O. Pearce @ 2007-10-17 1:35 UTC (permalink / raw)
To: Luke Lu; +Cc: git, pasky
Luke Lu <git@vicaya.com> wrote:
> Resubmit patch due to tab/space issue :)
Thanks, I have this locally from your prior version but already
had fixed the tab/space problem.
> +GITWEB_PROJECT_MAXDEPTH = 2007
Cute. But does what I was asking for, which was to not change
behavior for existing users. Most folks have a MAX_PATH around
1024-4096. There's no sane way they would exceed 2000 nested
directories.
--
Shawn.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitweb: speed up project listing by limiting find depth
2007-10-17 1:13 Luke Lu
2007-10-17 1:35 ` Shawn O. Pearce
@ 2007-10-17 2:40 ` Shawn O. Pearce
1 sibling, 0 replies; 6+ messages in thread
From: Shawn O. Pearce @ 2007-10-17 2:40 UTC (permalink / raw)
To: Luke Lu; +Cc: git, pasky
Luke Lu <git@vicaya.com> wrote:
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 3064298..d62357f 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -1509,16 +1513,23 @@ sub git_get_projects_list {
...
> + # don't traverse too deep (Find is super slow on os x)
> + if (tr!/!! - $pfxdepth > $project_maxdepth) {
> + $File::Find::prune = 1;
> + return;
> + }
>
> my $subdir = substr($File::Find::name, $pfxlen + 1);
Your patch appears to be causing some errors in the test suite
in t/t9500-gitweb-standalone-no-errors.sh. Perl is whining about
$subdir not getting initialized above due to the substr being off
the string. I've got too many other topics tonight to figure out
why yours is failing, can you please run the test and resubmit when
you've resolved the error?
--
Shawn.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] gitweb: speed up project listing by limiting find depth
@ 2007-10-17 0:24 Luke Lu
2007-10-17 0:41 ` Shawn O. Pearce
0 siblings, 1 reply; 6+ messages in thread
From: Luke Lu @ 2007-10-17 0:24 UTC (permalink / raw)
To: git; +Cc: gitster, Luke Lu
Resubmit patch to make project max depth configurable.
Signed-off-by: Luke Lu <git@vicaya.com>
---
Makefile | 2 ++
gitweb/gitweb.perl | 8 ++++++++
2 files changed, 10 insertions(+), 0 deletions(-)
diff --git a/Makefile b/Makefile
index 8db4dbe..b70ba8c 100644
--- a/Makefile
+++ b/Makefile
@@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl
GITWEB_HOME_LINK_STR = projects
GITWEB_SITENAME =
GITWEB_PROJECTROOT = /pub/git
+GITWEB_PROJECT_MAXDEPTH = 2
GITWEB_EXPORT_OK =
GITWEB_STRICT_EXPORT =
GITWEB_BASE_URL =
@@ -831,6 +832,7 @@ gitweb/gitweb.cgi: gitweb/gitweb.perl
-e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
-e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
-e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
+ -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
-e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
-e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
-e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 3064298..1453101 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -35,6 +35,10 @@ our $GIT = "++GIT_BINDIR++/git";
#our $projectroot = "/pub/scm";
our $projectroot = "++GITWEB_PROJECTROOT++";
+# fs traversing limit for getting project list
+# the number is relative to the projectroot
+our $project_maxdepth = "++GITWEB_PROJECT_MAXDEPTH++";
+
# target of the home link on top of all pages
our $home_link = $my_uri || "/";
@@ -1509,16 +1513,20 @@ sub git_get_projects_list {
# remove the trailing "/"
$dir =~ s!/+$!!;
my $pfxlen = length("$dir");
+ my $pfxdepth = ($dir =~ tr!/!!);
File::Find::find({
follow_fast => 1, # follow symbolic links
follow_skip => 2, # ignore duplicates
+ no_chdir => 1, # don't chdir into every directory
dangling_symlinks => 0, # ignore dangling symlinks, silently
wanted => sub {
# skip project-list toplevel, if we get it.
return if (m!^[/.]$!);
# only directories can be git repositories
return unless (-d $_);
+ # don't traverse too deep (Find is super slow on os x)
+ return if tr!/!! - $pfxdepth > $project_maxdepth && ($File::Find::prune = 1);
my $subdir = substr($File::Find::name, $pfxlen + 1);
# we check related file in $projectroot
--
1.5.3.4
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] gitweb: speed up project listing by limiting find depth
2007-10-17 0:24 Luke Lu
@ 2007-10-17 0:41 ` Shawn O. Pearce
0 siblings, 0 replies; 6+ messages in thread
From: Shawn O. Pearce @ 2007-10-17 0:41 UTC (permalink / raw)
To: Luke Lu; +Cc: git, gitster, Petr Baudis
Luke Lu <git@vicaya.com> wrote:
> diff --git a/Makefile b/Makefile
> index 8db4dbe..b70ba8c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl
> GITWEB_HOME_LINK_STR = projects
> GITWEB_SITENAME =
> GITWEB_PROJECTROOT = /pub/git
> +GITWEB_PROJECT_MAXDEPTH = 2
I'd rather see this default to an unlimited (or maybe insane?) depth.
Current users may be surprised upon upgrading to a more recent git
when their gitweb stops showing projects because the default depth
is too small.
repo.or.cz is up at 3 deep, maybe 4 right now, right Pasky?
I think letting admins control the depth is a good idea, but its
a performance tuning thing and probably shouldn't break existing
setups.
> + # don't traverse too deep (Find is super slow on os x)
> + return if tr!/!! - $pfxdepth > $project_maxdepth && ($File::Find::prune = 1);
I don't do much gitweb hacking, but I usually don't like to find
code that mutates a value as an important side-effect in the middle
of a boolean condition that is used to determine if we are breaking
out of this function now, or falling through to do more work. yea
its more lines of code but I think it would be easier to grok if this
was a proper if {...}.
--
Shawn.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2007-10-17 2:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-10-17 1:03 [PATCH] gitweb: speed up project listing by limiting find depth Luke Lu
-- strict thread matches above, loose matches on Subject: below --
2007-10-17 1:13 Luke Lu
2007-10-17 1:35 ` Shawn O. Pearce
2007-10-17 2:40 ` Shawn O. Pearce
2007-10-17 0:24 Luke Lu
2007-10-17 0:41 ` Shawn O. Pearce
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).