* Re: reexporting git repository via git-daemon
From: Jakub Narebski @ 2006-12-10 17:35 UTC (permalink / raw)
To: git
In-Reply-To: <200612082212.09682.arekm@maven.pl>
Arkadiusz Miskiewicz wrote:
> I have weird problem wit git (1.4.4.2).
I have tried to reproduce this error, but no luck.
> $ git --bare clone git://git.kernel.org/pub/scm/git/git.git
> fetches everything correctly;
It should be "git clone --bare git://git.kernel.org/pub/scm/git/git.git"
or "git clone --bare git://git.kernel.org/pub/scm/git/git.git git.git"
Git doesn't need GIT_DIR for clone.
> $ cd /tmp
> $ git clone /gitroot/home/gitrepo/git
>
> also correctly fetched. The problem begins with exporting that cloned repo
> once again via git-daemon:
I didn't do this second cloning, but it should not matter I think.
> $ git clone git://git.my-server/git
> fatal: unexpected EOF
> fetch-pack from 'git://git.my-server/git' failed.
True, the error messages of git-clone are bit cryptic and doesn't
give us much information. Does there exist such repository? Perhaps
it is not exported? Were there any error?
> strace on git-daemon side:
> [...]
> 18241 lstat("refs/tags/v1.4.1-rc2", {st_mode=S_IFREG|0644, st_size=41, ...}) =
> 0
> 18241 open("refs/tags/v1.4.1-rc2", O_RDONLY) = 7
> 18241 read(7, "33c9db007159db11c1ad5fa7101ea95853740acf\n", 255) = 41
> 18241 close(7) = 0
> 18241 write(1, "004233c9db007159db11c1ad5fa7101ea95853740acf
> refs/tags/v1.4.1-rc2\n", 66) = 66
> 18241 write(1, "0045abc02670169cee9541793a86324a014272ca8ed5
> refs/tags/v1.4.1-rc2^{}\n", 69) = 69
> 18241 stat("refs/tags/v1.4.1.1", {st_mode=S_IFREG|0644, st_size=41, ...}) = 0
> 18241 lstat("refs/tags/v1.4.1.1", {st_mode=S_IFREG|0644, st_size=41, ...}) = 0
> 18241 open("refs/tags/v1.4.1.1", O_RDONLY) = 7
> 18241 read(7, "8419a453dc088b25b63ab1746d3d7e679caf686d\n", 255) = 41
> 18241 close(7)
> 18241 write(1, "00408419a453dc088b25b63ab1746d3d7e679caf686d
> refs/tags/v1.4.1.1\n", 64) = 64
> 18241 write(2, "fatal: ", 7) = 7
> 18241 write(2, "corrupted pack
> file ./objects/pack/pack-0bb22295a585ac173504a2f8dfb3e31e074a8715.pack", 85)
> = 85
> 18241 write(2, "\n", 1)
>
> 100% repeatable at this moment. Any ideas?
WORKSFORME
$ git --version
git version 1.4.4.1
$ git clone --bare git://git.kernel.org/pub/scm/git/git.git
remote: Generating pack...
remote: Done counting 33587 objects.
remote: Deltifying 33587 objects.
remote: 100% (33587/33587) done
Indexing 33587 objects.
remote: Total 33587, written 33587 (delta 23205), reused 33325 (delta 23034)
100% (33587/33587) done
Resolving 23205 deltas.
100% (23205/23205) done
By the way, I wonder why git-clone names bare repository 'git',
and not 'git.git' as, I think, it should?
$ git-daemon --verbose --export-all --base-path=<pwd>
[31823] Connection from 127.0.0.1:46736
[31823] Extended attributes (16 bytes) exist <host=localhost>
[31823] Request upload-pack for '/git.git'
[31823] Disconnected
The above is generated when I was cloning locally, i.e. when I run the
following command:
$ git clone git://localhost/git.git
remote: Generating pack...
remote: Done counting 33587 objects.
remote: Deltifying 33587 objects.
remote: 100% (33587/33587) done
Indexing 33587 objects.
remote: Total 33587, written 33587 (delta 23205), reused 33587 (delta 23205)
100% (33587/33587) done
Resolving 23205 deltas.
100% (23205/23205) done
Checking files out...
100% (743/743) done
which went without any problem.
P.S. Please reply also to git mailing list
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: reexporting git repository via git-daemon
From: Arkadiusz Miskiewicz @ 2006-12-10 17:43 UTC (permalink / raw)
To: git
In-Reply-To: <elhgcf$m4r$1@sea.gmane.org>
On Sunday 10 December 2006 18:35, Jakub Narebski wrote:
> Arkadiusz Miskiewicz wrote:
> > I have weird problem wit git (1.4.4.2).
>
> I have tried to reproduce this error, but no luck.
Now I tracked that down to git-daemon in version 1.4.2.4 not 1.4.4.2. Seems
that something was later fixed, so 1.4.4.2 is OK.
|
--
Arkadiusz Miśkiewicz PLD/Linux Team
^ permalink raw reply
* Re: Using GIT to store /etc (Or: How to make GIT store all file permission bits)
From: Kyle Moffett @ 2006-12-10 17:46 UTC (permalink / raw)
To: Santi Béjar, Jeff Garzik; +Cc: git
In-Reply-To: <8aa486160612100706y92bc722n93374e394fc58005@mail.gmail.com>
> On 12/10/06, Kyle Moffett <mrmacman_g4@mac.com> wrote:
>> I've recently become somewhat interested in the idea of using GIT
>> to store the contents of various folders in /etc. However after a
>> bit of playing with this, I discovered that GIT doesn't actually
>> preserve all permission bits since that would cause problems with
>> the more traditional software development model. I'm curious if
>> anyone has done this before; and if so, how they went about
>> handling the permissions and ownership issues.
>>
>> I spent a little time looking over how GIT stores and compares
>> permission bits; trying to figure out if it's possible to patch in
>> a new configuration variable or two; say "preserve_all_perms" and
>> "preserve_owner", or maybe even "save_acls". It looks like
>> standard permission preservation is fairly basic; you would just
>> need to patch a few routines which alter the permissions read in
>> from disk or compare them with ones from the database. On the
>> other hand, it would appear that preserving ownership or full
>> POSIX ACLs might be a bit of a challenge.
On Dec 10, 2006, at 10:06:14, Santi Béjar wrote:
> I have not used it, but you could try:
>
> http://www.isisetup.ch/
>
> that uses git as a backend.
Wow, umm, that's actually really interesting for me, given that I'm
most interested in these sorts of things on Debian. I can't find
much documentation on their site; the tools look vaguely immature but
I haven't really had much time to look at it yet.
On Dec 10, 2006, at 09:49:50, Jeff Garzik wrote:
> It's a great idea, something I would like to do, and something I've
> suggested before. You could dig through the mailing list archives,
> if you're motivated.
I have been digging through the archives; I was just holding out hope
that somebody else on the list had already halfway beat me to the
punch. Guess not :-D
> I actively use git to version, store and distribute an exim mail
> configuration across six servers. So far my solution has been a
> 'fix perms' script, or using the file perm checking capabilities of
> cfengine.
>
> But it would be a lot better if git natively cared about ownership
> and permissions (presumably via an option).
I was thinking about a standard config option in the GIT config file,
that way users could have a personal default and repositories could
specify it locally.
I started tinkering but quickly discovered that permissions handling
in general in GIT seems to be a mess; there's about 4 different tiers
where permissions data is manipulated in various formats. Some
places use network-endian 16-bit values, there's a couple functions
which do different truncations to 644 or 755 format. There are 2
functions which canonicalize the file mode based on symlink or
directory status, each in subtly different ways.
I'm slowly sorting through things but if I could get a few pointers
from someone intimately familiar with the code that would be most
appreciated: I'd like to try to add new entries to tree objects
which older versions of GIT would ignore but which newer versions of
GIT would use to store ACL or extended-attribute data.
The simplest solution which admittedly breaks the ability of older
GITs to read the data from a file with attributes (ignoring the ext-
attrs themselves) is to create a new "file-with-extended-attributes"
object which contains a binary concatenation (with length bytes and
attribute names and such) of the file and its extended attributes.
That breaks the old GIT assumption that permission and security data
is part of the directory not the file, but it's more in-line with the
way extended attributes are attached to the inodes in the filesystem
(although that doesn't really matter IMO).
Alternatively I might be able to add a new entry to each tree object
with invalid extended file mods bits (IE: Neither a directory, a
file, nor a symlink), or perhaps an entry with an empty name, which
points to a new "extended attribute table". That table could either
map from (entry, attribute) => (data) or from (entry) =>
((attribute,data),(attribute,data),[...]), depending on which would
be more efficient. It's essential that the overhead for non-ext-attr
repositories is O(1) and ideally the overhead for a bunch of files
with the same ext-attr is O(size-of-ext-attr) + O(number-of-files-
with-that-attr), although that may vary depending on implementation.
Advice, opinions, problems, and "this-has-no-chance-of-ever-even-
remotely-working" are all useful and welcome!
Cheers,
Kyle Moffett
^ permalink raw reply
* Re: Using GIT to store /etc (Or: How to make GIT store all file permission bits)
From: Jakub Narebski @ 2006-12-10 18:10 UTC (permalink / raw)
To: git
In-Reply-To: <28E2300C-8F7A-406F-8FDA-F8786AE95B40@mac.com>
Kyle Moffett wrote:
> The simplest solution which admittedly breaks the ability of older
> GITs to read the data from a file with attributes (ignoring the ext-
> attrs themselves) is to create a new "file-with-extended-attributes"
> object which contains a binary concatenation (with length bytes and
> attribute names and such) of the file and its extended attributes.
> That breaks the old GIT assumption that permission and security data
> is part of the directory not the file, but it's more in-line with the
> way extended attributes are attached to the inodes in the filesystem
> (although that doesn't really matter IMO).
This contradict git philosophy of "tracking contents".
> Alternatively I might be able to add a new entry to each tree object
> with invalid extended file mods bits (IE: Neither a directory, a
> file, nor a symlink), or perhaps an entry with an empty name, which
> points to a new "extended attribute table". That table could either
> map from (entry, attribute) => (data) or from (entry) =>
> ((attribute,data),(attribute,data),[...]), depending on which would
> be more efficient. It's essential that the overhead for non-ext-attr
> repositories is O(1) and ideally the overhead for a bunch of files
> with the same ext-attr is O(size-of-ext-attr) + O(number-of-files-
> with-that-attr), although that may vary depending on implementation.
Wouldn't it be better to add another field in the tree object, that
instead of storing "(filemode, link to contents, name)" it would
store "(filemode, link to extended attributes, link to contents, name)"
where "filemode" is mode of a file of which git uses only a few bits
(is a directory, is a symlink, is a file, is a executable file),
and "link to" is sha1 of appropriate blob (or tree) object? Extended
attributes could be stored in new type of object, or just in blob
object. Well, you'd have to extend index in similar way (and add
a way to store extended attributes for directories in index; nowit only
stores info about files).
This of course breaks backwards compatibility...
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Using GIT to store /etc (Or: How to make GIT store all file permission bits)
From: Kyle Moffett @ 2006-12-10 18:10 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <elh91b$v6r$1@sea.gmane.org>
On Dec 10, 2006, at 10:30:00, Jakub Narebski wrote:
> Jeff Garzik wrote:
>> I actively use git to version, store and distribute an exim mail
>> configuration across six servers. So far my solution has been a
>> 'fix perms' script, or using the file perm checking capabilities
>> of cfengine.
>
> Fix perms' script used on a checkout hook is a best idea I think.
Hmm, unfortunately that has problems with security-related race
conditions when used directly for /etc. Think about what happens
with "/etc/shadow" in that case, for example. (/etc/.git is of
course 0700) I'm sure there are others where non-root daemons get
unhappy when they get an inotify event and their config files have
suddenly become root:root:0600. I also want to be able to "cd /etc
&& git status" to see what changed after running "apt-get update" or
maybe fiddling in SWAT or webmin, so a makefile which installs into /
etc won't quite solve it either. It would also be nice to see when
things change the permissions on files in /etc, or even bind-mount an
append-only volume over /etc/.git/objects to provide additional data
security.
>> But it would be a lot better if git natively cared about ownership
>> and permissions (presumably via an option).
>
> There is currently no place for ownership and extended attributes
> in the tree object; and even full POSIX permissions might be
> challenge because for example currently unused 'is socket'
> permission bit is used for experimental commit-in-tree submodule
> support.
What about doing something crazy like "is socket" && "is directory"
&& "is symlink"? Or something else that old GIT versions would
ignore and new GIT versions could do something useful with? Perhaps
like I mentioned in an earlier email, the new data could be stored as
part of a modified "file" object. Alternatively could a directory
have a file named with an empty string with bogus mode bits which
points to an extended-attributes-tree object?
> And given Linus stance that git is "content tracker"...
Extended attributes are content too! This includes things like
icons, security labels (Think unclassified/confidential/secret/top-
secret/etc), ACLs, summaries, and other metadata. Content tracker
purists could also just ignore the new default-off config options and
be perfectly happy with status-quo. :-D
> In the loooong thread "VCS comparison table" there was some talk
> about using git (or any SCM) to manage /etc. Check out:
>
> * Message-ID: <Pine.LNX.4.64.0610220926170.3962@g5.osdl.org>
> http://permalink.gmane.org/gmane.comp.version-control.git/29765
> * Message-ID: <20061023051932.GA8625@evofed.localdomain>
> http://marc.theaimsgroup.com/?
> i=<20061023051932.GA8625@evofed.localdomain>
>
> (and other messages in this subthread).
I have, and while it's interesting material that thread produced no
real patches :-D. I'd like to introduce some new config options to
control the new code: "preserve_full_perms", "preserve_posix_acls",
"preserve_security_labels", and "preserve_user_xattrs" which default
to false but when set modify GIT's behavior to store, retrieve, and
compare additional data.
If you have any suggestions on how to store the data such that old
GIT ignores it I'm all ears :-D.
Cheers,
Kyle Moffett
^ permalink raw reply
* Re: Using GIT to store /etc (Or: How to make GIT store all file permission bits)
From: Jakub Narebski @ 2006-12-10 18:18 UTC (permalink / raw)
To: git
In-Reply-To: <A52817B6-0265-4164-8E5D-334AF92DC267@mac.com>
Kyle Moffett wrote:
> On Dec 10, 2006, at 10:30:00, Jakub Narebski wrote:
>> Jeff Garzik wrote:
>>>
>>> I actively use git to version, store and distribute an exim mail
>>> configuration across six servers. So far my solution has been a
>>> 'fix perms' script, or using the file perm checking capabilities
>>> of cfengine.
>>
>> Fix perms' script used on a checkout hook is a best idea I think.
>
> Hmm, unfortunately that has problems with security-related race
> conditions when used directly for /etc. Think about what happens
> with "/etc/shadow" in that case, for example. (/etc/.git is of
> course 0700) I'm sure there are others where non-root daemons get
> unhappy when they get an inotify event and their config files have
> suddenly become root:root:0600. I also want to be able to "cd /etc
> && git status" to see what changed after running "apt-get update" or
> maybe fiddling in SWAT or webmin, so a makefile which installs into /
> etc won't quite solve it either. It would also be nice to see when
> things change the permissions on files in /etc, or even bind-mount an
> append-only volume over /etc/.git/objects to provide additional data
> security.
The idea is to not store /etc in git directly, but use import/export
scripts, which for example saves permissions and ownership in some
file also tracked by git on import, and restores correct permissions
on export. That is what I remember from this discussion. This of course
means that you would have to write your own porcelain...
What about mentioned in other email IsiSetup?
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [RFC \ WISH] Add -o option to git-rev-list
From: Linus Torvalds @ 2006-12-10 18:16 UTC (permalink / raw)
To: Marco Costalba
Cc: Git Mailing List, Junio C Hamano, Alex Riesen, Shawn Pearce
In-Reply-To: <e5bfff550612100338ye2ca2a0u1c8f29bbc59c5431@mail.gmail.com>
On Sun, 10 Dec 2006, Marco Costalba wrote:
>
> The averaged results on linux tree (about 30MB of data) and CPU set at
> 1.2GHz are:
>
> - QProcess 6734ms
>
> - pipe and fread() with 64KB blocks 4832ms (38% faster then QProcess)
>
> - temporary file and read() with 64KB blocks 4321ms (10% faster then pipe)
>
> I have not enough knowledge to understand why temporary file is faster
> then pipe. My guess is, after reading some docs around, fread() uses a
> C standard I/O buffer, while read() is unbuffered.
Why don't you use the pipe and standard read()?
Even if you use "popen()" and get a "FILE *" back, you can still do
int fd = fileno(file);
and use the raw IO capabilities.
The thing is, temporary files can actually be faster under Linux just
because the Linux page-cache simply kicks ass. But it's not going to be
_that_ big of a difference, and you need all that crazy "wait for rev-list
to finish" and the "clean up temp-file on errors" etc crap, so there's no
way it's a better solution.
If it really is stdio overhead (possibly locking), using "fileno()" and
the raw unistd.h interfaces is going to avoid it.
(You still need to use "fclose()" to close the struct file afterwards,
otherwise you'll leak memory, so you shouldn't _forget_ the original
"struct FILE *", but you don't need to use it for anything else).
Using popen() and pipes also means that if the parent dies, the child will
get a nice EPIPE on the writing side, which is what you want.
^ permalink raw reply
* Re: Using GIT to store /etc (Or: How to make GIT store all file permission bits)
From: Jakub Narebski @ 2006-12-10 18:26 UTC (permalink / raw)
To: Kyle Moffett; +Cc: git
In-Reply-To: <A52817B6-0265-4164-8E5D-334AF92DC267@mac.com>
Kyle Moffett wrote:
> On Dec 10, 2006, at 10:30:00, Jakub Narebski wrote:
>> Jeff Garzik wrote:
>>>
>>> I actively use git to version, store and distribute an exim mail
>>> configuration across six servers. So far my solution has been a
>>> 'fix perms' script, or using the file perm checking capabilities
>>> of cfengine.
>>
>> Fix perms' script used on a checkout hook is a best idea I think.
>
> Hmm, unfortunately that has problems with security-related race
> conditions when used directly for /etc. Think about what happens
> with "/etc/shadow" in that case, for example. (/etc/.git is of
> course 0700) I'm sure there are others where non-root daemons get
> unhappy when they get an inotify event and their config files have
> suddenly become root:root:0600. I also want to be able to "cd /etc
> && git status" to see what changed after running "apt-get update" or
> maybe fiddling in SWAT or webmin, so a makefile which installs into /
> etc won't quite solve it either. It would also be nice to see when
> things change the permissions on files in /etc, or even bind-mount an
> append-only volume over /etc/.git/objects to provide additional data
> security.
The idea is to not store /etc in git directly, but use import/export
scripts, which for example saves permissions and ownership in some
file also tracked by git on import, and restores correct permissions
on export. That is what I remember from this discussion. This of course
means that you would have to write your own porcelain...
What about mentioned in other email IsiSetup?
--
Jakub Narebski
Warsaw, Poland
^ permalink raw reply
* Re: Using GIT to store /etc (Or: How to make GIT store all file permission bits)
From: Kyle Moffett @ 2006-12-10 18:35 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <200612101926.33307.jnareb@gmail.com>
On Dec 10, 2006, at 13:26:32, Jakub Narebski wrote:
> The idea is to not store /etc in git directly, but use import/
> export scripts, which for example saves permissions and ownership
> in some file also tracked by git on import, and restores correct
> permissions on export. That is what I remember from this
> discussion. This of course means that you would have to write your
> own porcelain...
>
> What about mentioned in other email IsiSetup?
The real problem I have with that is you literally have to duplicate
all sorts of functionality. I want to run "foo-status" in /etc and
get something useful, but if /etc is not a git directory in and of
itself then you have to duplicate most of "git-status" anyways. And
the same applies to all the other commands. From what I can see of
IsiSetup the tools for checking out, merging, modifying, cloning, etc
are all much more limited and immature than the ones available
through GIT/cogito, and I would be loathe to discard all that extra
functionality and duplicate a few thousand lines of code in the name
of "concept purity".
GIT already has _some_ idea about file permissions, it just discards
most of the data before writing to disk. Of course, adding POSIX
ACLs and user-extended-attributes requires a new data format, but
those are very similar to filesystem permissions; they differ only in
amount of data stored, not in purpose.
Import/export scripts literally require wrapping every single GIT
command with a script that changes directory a few times, reads from
a different checked-out tree, and permutes some extended-attribute
data slightly before storing it in the underlying GIT tree. Even
without adding any new functionality whatsoever that doubles the
amount of code just for finding your repository and checking command-
line arguments, and that's a crazy trade-off to make in any situation.
Cheers,
Kyle Moffett
^ permalink raw reply
* Re: [PATCH 2/3] merge-recursive: make empty tree a known object
From: Linus Torvalds @ 2006-12-10 18:37 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Shawn O. Pearce, Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.63.0612100055390.28348@wbgn013.biozentrum.uni-wuerzburg.de>
On Sun, 10 Dec 2006, Johannes Schindelin wrote:
>
> To use it in diff_tree_sha1(), a tree has to be hashed in the
> global object collection. This actually moves the empty tree (if
> it is needed) into the global object hash.
I think you should do this more generically.
If we create this kind of "default fake object" that we know about whether
the object _really_ exists or not (and I agree it makes sense for the
empty tree), we should probably use it for "diff_root_tree_sha1()" too,
rather than the special case we have now.
In other words, right now we have that very special
"diff_root_tree_sha1()" function, but if you create a generic fake "empty
tree" SHA1 that git knows about implicitly, we could entirely replace it
with just using
diff_tree_sha1(EMPTY_TREE_SHA1, tree, base, opt);
instead, and get rid of that special case code (which is efficient, but we
don't really _need_ the efficiency).
So you could make "read_sha1_file()" just have a special case for known
objects at the end. If the pack entry fails, the loose file case fails,
then rather than returning NULL at the end, you could have a list of known
fixed objects..
Hmm?
^ permalink raw reply
* Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
From: Jakub Narebski @ 2006-12-10 19:11 UTC (permalink / raw)
To: Jeff Garzik
Cc: Martin Langhoff, Git Mailing List, Linus Torvalds, H. Peter Anvin,
Rogan Dawes, Kernel Org Admin
In-Reply-To: <457C0F8F.7030504@garzik.org>
Jeff Garzik wrote:
> Jakub Narebski wrote:
>> Jeff Garzik wrote:
>>> Jakub Narebski wrote:
>>>>
>>>> P.S. Any hints to how to do this with CGI Perl module?
>>>
>>> It's impossible, Apache doesn't supply e-tag info to CGI programs. (it
>>> does supply HTTP_CACHE_CONTROL though apparently)
>>
>> By ETag info you mean access to HTTP headers sent by browser
>> If-Modified-Since:, If-Match:, If-None-Match: do you?
Adn in CGI standard there is a way to access additional HTTP headers
info from CGI script: the envirionmental variables are HTTP_HEADER,
for example if browser sent If-Modified-Since: header it's value
can be found in HTTP_IF_MODIFIED_SINCE environmental variable.
But of course gitweb should rather use mod_perl if possible, so
somewhere in gitweb there would be the following line:
$in_date = $ENV{'MOD_PERL'} ?
$r->header('If-Modified-Since') :
$ENV{'HTTP_IF_MODIFIED_SINCE'};
or something like that...
> You can use this attached shell script as a CGI script, to see precisely
> what information Apache gives you. You can even experiment with passing
> back headers other than Content-type (such as E-tag), to see what sort
> of results are produced. The script currently passes back both E-Tag
> and Last-Modified of a sample file; modify or delete those lines to suit
> your experiments.
It is ETag, not E-tag. Besides, I don't see what the attached script is
meant to do: it does not output the sample file anyway.
>> It's a pity that CGI interface doesn't cover that...
>>
>>> You could probably do it via mod_perl.
>>
>> So the cache verification should be wrapped in if ($ENV{MOD_PERL}) ?
>
> Sorry, I was /assuming/ mod_perl would make this available. The HTTP
> header info is available to all Apache modules, but I confess I have no
> idea how mod_perl passes that info to scripts.
>
> Also, an interesting thing while I was testing the attached shell
> script: even though repeated hits to the script generate a proper 304
> response to the browse, the CGI script and its output run to completion.
> So, it didn't save work on the CGI side; the savings was solely in not
> transmitting the document from server to client. The server still went
> through the work of generating the document (by running the CGI), as one
> would expect.
The idea is of course to stop processing in CGI script / mod_perl script
as soon as possible if cache validates.
I don't know if Apache intercepts and remembers ETag and Last-Modified
headers, adds 304 Not Modified HTTP response on finding that cache validates
and cuts out CGI script output. I.e. if browser provided If-Modified-Since:,
script wrote Last-Modified: header, If-Modified-Since: is no earlier than
Last-Modified: (usually is equal in the case of cache validation), then
Apache provides 304 Not Modified response instead of CGI script output.
--
Jakub Narebski
^ permalink raw reply
* Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
From: Linus Torvalds @ 2006-12-10 19:50 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jeff Garzik, Martin Langhoff, Git Mailing List, H. Peter Anvin,
Rogan Dawes, Kernel Org Admin
In-Reply-To: <200612102011.52589.jnareb@gmail.com>
On Sun, 10 Dec 2006, Jakub Narebski wrote:
> >> If-Modified-Since:, If-Match:, If-None-Match: do you?
>
> Adn in CGI standard there is a way to access additional HTTP headers
> info from CGI script: the envirionmental variables are HTTP_HEADER,
> for example if browser sent If-Modified-Since: header it's value
> can be found in HTTP_IF_MODIFIED_SINCE environmental variable.
Guys, you're missing something fairly fundamnetal.
It helps almost _nothing_ to support client-side caching with all these
fancy "If-Modified-Since:" etc crap.
That's not the _problem_.
It's usually not one client asking for the gitweb pages: the load comes
from just lots of people independently asking for it. So client-side
caching may help a tiny tiny bit, but it's not actually fixing the
fundamental problem at all.
So forget about "If-Modified-Since:" etc. It may help in benchmarks when
you try it yourself, and use "refresh" on the client side. But the basic
problem is all about lots of clients that do NOT have things cached,
because all teh client caches are all filled up with pr0n, not with gitweb
data from yesterday.
So the thing to help is server-side caching with good access patterns, so
that the server won't have to seek all over the disk when clients that
_don't_ have things in their caches want to see the "git projects" summary
overview (that currently lists something like 200+ projects).
So to get that list of 200+ projects, right now gitweb will literally walk
them all, look at their refs, their descriptions, their ages (which
requires looking up the refs, and the objects behing the refs), and if
they aren't cached, you're going to have several disk seeks for each
project.
At 200+ projects, the thing that makes it slow is those disk seeks. Even
with a fast disk and RAID array, the seeks are all basically going to be
interdependent, so there's no room for disk arm movement optimization, and
in the absense of any other load it's still going to be several seconds
just for the seeks (say 10ms per seek, four or five seeks per project,
you've got 10 seconds _just_ for the seeks to generate the top-level
summary page, and quite frankly, five seeks is probably optimistic).
Now, hopefully some of it will be in the disk cache, but when the
mirroring happens, it will basically blow the disk caches away totally
(when using the "--checksum" option), and then you literally have tens of
seconds to generate that one top-level page.
And when mirroring is blowing out the disk caches, the thing will be doing
other things _too_ to the disk, of course.
So what you want is server-side caching, and you basically _never_ want to
re-generate that data synchronously (because even if the server can take
the load, having the clients wait for half a minute or more for the data
is just NOT FRIENDLY). This is why I suggested the grace-period where we
fill the cache on he server side in the background _while_at_the_same_time
actually feeding the clients the old cached contents.
Because what matters most to _clients_ is not getting the most recent
up-to-date data within the last few minutes - people who go to the
overview page want to just get a list of projects, and they want to get
them in a second or two, not half a minute later.
And btw, all those "If-Modified-Since:" things are irrelevant, since quite
often, the top-level page really technically _has_ been modified in the
last few minutes, because with the kernel and git projects, _somebody_ has
usually pushed out one of the projects within the last hour.
And no, people don't just sit there refreshing their browser page all the
time. I bet even "active" git users do it at most once or twice a day,
which means that their client cache will _never_ be up-to-date.
But if you do it with server-side caches and grace-periods, you can
generally say "we have something that is at most five minutes old", and
most importantly, you can hopefully do it without a lot of disk seeks
(because you just cache the _one_ page as _one_ object), so hopefully you
can do it in a few hundred ms even if the thing is on disk and even if
there's a lot of other load going on.
I bet the top-level "all projects" summary page and the individual
project summary pages are the important things to cache. That's what
probably most people look at, and they are the ones that have lots of
server-side cache locality. Individual commits and diffs probably don't
get the same kind of "lots of people looking at them" and thus don't get
the same kind of benefit from caching.
(Individual commits hopefully also need fewer disk seeks, at least with
packed repositories. So even if you have to re-generate them from scratch,
they won't have the seek times themselves taking up tens of seconds,
unless the project is entirely unpacked and diffing just generates total
disk seek hell)
^ permalink raw reply
* Re: [RFC \ WISH] Add -o option to git-rev-list
From: Marco Costalba @ 2006-12-10 19:51 UTC (permalink / raw)
To: Linus Torvalds
Cc: Git Mailing List, Junio C Hamano, Alex Riesen, Shawn Pearce
In-Reply-To: <Pine.LNX.4.64.0612101009040.12500@woody.osdl.org>
On 12/10/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
> Why don't you use the pipe and standard read()?
>
> Even if you use "popen()" and get a "FILE *" back, you can still do
>
> int fd = fileno(file);
>
> and use the raw IO capabilities.
>
> The thing is, temporary files can actually be faster under Linux just
> because the Linux page-cache simply kicks ass. But it's not going to be
> _that_ big of a difference, and you need all that crazy "wait for rev-list
> to finish" and the "clean up temp-file on errors" etc crap, so there's no
> way it's a better solution.
>
Two things.
- memory use: the next natural step with files is, instead of loading
the file content in memory and *keep it there*, we could load one
chunk at a time, index the chunk and discard. At the end we keep in
memory only indexing info to quickly get to the data when needed, but
the big part of data stay on the file.
- This is probably my ignorance, but experimenting with popen() I
found I could not know *when* git-rev-list ends because both feof()
and ferror() give 0 after a fread() with git-rev-list already defunct.
Not having a reference to the process (it is hidden behind popen() ),
I had to check for 0 bytes read after a successful read (to avoid
racing in case I ask the pipe before the first data it's ready) to
know that job is finished and call pclose().
^ permalink raw reply
* globs in partial checkout?
From: Michael S. Tsirkin @ 2006-12-10 20:00 UTC (permalink / raw)
To: Git Mailing List, Junio C Hamano
In-Reply-To: <e5bfff550612101151r5e77ddc0w8f61f36fc1f2920e@mail.gmail.com>
I'm trying to checkout some files after doing "clone -n".
Should using globs there work? It doesn't:
st@mst-lt:~/scm/wireless-dev$ git checkout master 'include/net/ieee80211*.h'
error: pathspec 'include/net/ieee80211*.h' did not match any file(s) known to
git.
Did you forget to 'git add'?
mst@mst-lt:~/scm/wireless-dev$ git ls-tree master -- include/net/ | grep iee
100644 blob b174ebb277a96668f058e469b0753503c34f164b include/net/ieee80211.h
100644 blob eb476414fd726701d032e9e517751b9d3f7e38df include/net/ieee80211_crypt.h
100644 blob 429b73892a5fc62f91e4a4b05da40859604fa791 include/net/ieee80211_radiotap.h
100644 blob 617b672b1132e7fa3ff5f9c940b1692520dc8483 include/net/ieee80211softmac.h
100644 blob 4ee3ad57283fa3370bd2d1f71cd6ae559b556dbc include/net/ieee80211softmac_wx.h
mst@mst-lt:~/scm/wireless-dev$ git checkout master include/net/ieee80211.h
include/net/ieee80211_crypt.h include/net/ieee80211_radiotap.h
include/net/ieee80211softmac.h include/net/ieee80211softmac_wx.h
--
^ permalink raw reply
* Re: [RFC \ WISH] Add -o option to git-rev-list
From: Linus Torvalds @ 2006-12-10 20:08 UTC (permalink / raw)
To: Marco Costalba
Cc: Git Mailing List, Junio C Hamano, Alex Riesen, Shawn Pearce
In-Reply-To: <e5bfff550612101151r5e77ddc0w8f61f36fc1f2920e@mail.gmail.com>
On Sun, 10 Dec 2006, Marco Costalba wrote:
>
> - memory use: the next natural step with files is, instead of loading
> the file content in memory and *keep it there*, we could load one
> chunk at a time, index the chunk and discard. At the end we keep in
> memory only indexing info to quickly get to the data when needed, but
> the big part of data stay on the file.
Well, that's still just going slower than swapping.
The reason loading things into memory is nice is that:
- in the common case, you don't need to do anything else.
- if the machine is low on memory, it can page things out just about as
easily as you could write things to a file anyway.
So don't worry too much about low-memory situations. Yes, there are cases
where it's better to keep things in files and simply not have a big
working set AT ALL, but if you keep something in a file and the file data
is still part of the working set (ie you read it several times, but at the
beginning and the end), that really isn't any better than having it in
memory.
So the time to try to optimize memory usage is really only for "streaming
behaviour" - where you need to touch something only once. Then the best
option is to actually use a pipe and re-use the memory, but if you have
file data, you can use things like fadvise(DONTNEED).
> - This is probably my ignorance, but experimenting with popen() I
> found I could not know *when* git-rev-list ends because both feof()
> and ferror() give 0 after a fread() with git-rev-list already defunct.
I suspect you had a bug somewhere. It could be a bug in stdio, but I doubt
it.
You do realize that the correct way to check "feof()" is only _after_
fread() returns 0?
Stdio ferror/feof is horrible to use corrrectly, and few people get it
right. Mostly because it's such a crap interface thanks to being "easy"
to use and thus hiding all the error handling details on purpose.
^ permalink raw reply
* Re: globs in partial checkout?
From: Linus Torvalds @ 2006-12-10 20:13 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Git Mailing List, Junio C Hamano
In-Reply-To: <20061210200029.GF1708@mellanox.co.il>
On Sun, 10 Dec 2006, Michael S. Tsirkin wrote:
>
> I'm trying to checkout some files after doing "clone -n".
> Should using globs there work? It doesn't:
Not historically at all. "git checkout" needed exact filenames in older
versions.
However, since about 1.4.4.1 or so, it now does the same filename
expansion as "git add" etc does, which means that you can give it a
directory name and it will check out everything under that directory, or
you can give it a pattern, and it should glob it. But it sounds like you
may have a slightly older version of git (the pathname matching really is
fairly recent).
^ permalink raw reply
* Re: [RFC \ WISH] Add -o option to git-rev-list
From: Linus Torvalds @ 2006-12-10 20:19 UTC (permalink / raw)
To: Marco Costalba
Cc: Git Mailing List, Junio C Hamano, Alex Riesen, Shawn Pearce
In-Reply-To: <Pine.LNX.4.64.0612101159530.12500@woody.osdl.org>
On Sun, 10 Dec 2006, Linus Torvalds wrote:
>
> So don't worry too much about low-memory situations.
Btw, I should obviously clarify that. You should _always_ worry about
low-memory situations, but the only real issue is really "working set
size", not "data in files or in memory".
If the file representation is very dense (like the git pack-files are, for
example), then it may well make sense to keep the data in a file, just
because it's smaller there than if you keep it in expanded form in memory.
Also, it's nice to keep stuff in the filesystem rather than in process VM
memory, because filesystem data that is cached in memory is useful for
_other_ processes, ie it has a lifetime that is longer than the process
itself. However, that's obviously only true for long-lived files that are
shared among processes, it's _not_ true for temporary files.
For temporary files, the memory footprint of a temp-file is usually
_larger_ than the memory footprint of the same data kept in memory. You
have things like page-cache alignment, and often the issue of marshalling
data into ASCII etc. So temp-files are almost never a better solution than
keeping things in memory (unless you use those temp-files to truly
_share_ data between processes, ie you do a shared mmap and they can
re-use the same pages actively in a way they couldn't otherwise).
^ permalink raw reply
* Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
From: Jakub Narebski @ 2006-12-10 20:27 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jeff Garzik, Martin Langhoff, Git Mailing List, H. Peter Anvin,
Rogan Dawes, Kernel Org Admin
In-Reply-To: <Pine.LNX.4.64.0612101129190.12500@woody.osdl.org>
Linus Torvalds wrote:
> On Sun, 10 Dec 2006, Jakub Narebski wrote:
>>>> If-Modified-Since:, If-Match:, If-None-Match: do you?
>>
>> And in CGI standard there is a way to access additional HTTP headers
>> info from CGI script: the envirionmental variables are HTTP_HEADER,
>> for example if browser sent If-Modified-Since: header it's value
>> can be found in HTTP_IF_MODIFIED_SINCE environmental variable.
>
> Guys, you're missing something fairly fundamnetal.
>
> It helps almost _nothing_ to support client-side caching with all these
> fancy "If-Modified-Since:" etc crap.
>
> That's not the _problem_.
>
> It's usually not one client asking for the gitweb pages: the load comes
> from just lots of people independently asking for it. So client-side
> caching may help a tiny tiny bit, but it's not actually fixing the
> fundamental problem at all.
Well, the idea (perhaps stupid idea: I don't know how caching engines
/ reverse proxy works) was that there would be caching engine / reverse
proxy in the front (Squid for example) would cache results and serve it
to rampaging hordes. But this caching engine has to ask gitweb if the
cache is valid using "If-Modified-Since:" and "If-None-Match:" headers.
If gitweb returns 304 Not Modified then it serves contents from cache.
> So forget about "If-Modified-Since:" etc. It may help in benchmarks when
> you try it yourself, and use "refresh" on the client side. But the basic
> problem is all about lots of clients that do NOT have things cached,
> because all teh client caches are all filled up with pr0n, not with gitweb
> data from yesterday.
What about the other idea, the one with raising expires to infinity for
immutable pages like "commit" view for commit given by SHA-1? Even if
the clients won't cache it, the proxies and caches between gitweb and
client might cache it...
Talking about most accessed gitweb pages, the project list page changes
on every push, the project summary page and project main RSS feed
(now in both RSS and Atom formats) changes on every push to given project.
With a help of hooks they can be static pages, generated by push...
...with the exception that projects list and summary pages have _relative_
dates.
--
Jakub Narebski
^ permalink raw reply
* Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
From: Linus Torvalds @ 2006-12-10 20:30 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jeff Garzik, Martin Langhoff, Git Mailing List, H. Peter Anvin,
Rogan Dawes, Kernel Org Admin
In-Reply-To: <200612102127.05894.jnareb@gmail.com>
On Sun, 10 Dec 2006, Jakub Narebski wrote:
>
> Well, the idea (perhaps stupid idea: I don't know how caching engines
> / reverse proxy works) was that there would be caching engine / reverse
> proxy in the front (Squid for example) would cache results and serve it
> to rampaging hordes.
Sure, if the proxies actually do the rigth thing (which they may or may
not do)
> What about the other idea, the one with raising expires to infinity for
> immutable pages like "commit" view for commit given by SHA-1? Even if
> the clients won't cache it, the proxies and caches between gitweb and
> client might cache it...
I agree, but as mentioned, I think the _real_ problem tends to be the
pages that don't act that way (ie summary pages, both at the individual
project level and the top "all projects" level).
^ permalink raw reply
* [RFC/PATCH] gitk: add current directory to main window title
From: Doug Maxey @ 2006-12-10 20:31 UTC (permalink / raw)
To: Paul Mackerras; +Cc: Git Mailing List
From fe613c56b0079871e6b862077a9d673ca064cc16 Mon Sep 17 00:00:00 2001
From: Doug Maxey <dwm@bebe.enoyolf.org>
Date: Sun, 10 Dec 2006 14:02:50 -0600
Signed-off-by: Doug Maxey <dwm@enoyolf.org>
---
Based off gitk#master.
It helps me, being greyware challenged, keep track of which gitk is active
for a given directory.
---
gitk | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/gitk b/gitk
index 3dabc69..b8fd6ca 100755
--- a/gitk
+++ b/gitk
@@ -6293,6 +6293,7 @@ set stuffsaved 0
set patchnum 0
setcoords
makewindow
+wm title . "[file tail $argv0] [file tail [pwd]]"
readrefs
if {$cmdline_files ne {} || $revtreeargs ne {}} {
--
1.4.4.2
^ permalink raw reply related
* Re: [PATCH] Documentation/git-commit: rewrite to make it more end-user friendly.
From: J. Bruce Fields @ 2006-12-10 21:00 UTC (permalink / raw)
To: Josef Weidendorfer; +Cc: Junio C Hamano, Nicolas Pitre, git
In-Reply-To: <200612100130.48812.Josef.Weidendorfer@gmx.de>
On Sun, Dec 10, 2006 at 01:30:47AM +0100, Josef Weidendorfer wrote:
> On Saturday 09 December 2006 06:48, Junio C Hamano wrote:
> > +1. by using gitlink:git-add[1] to incrementally "add" changes to the
> > + next commit before using the 'commit' command (Note: even modified
> > + files must be "added");
>
> Regarding this note: Of course unmodified files do not have to be added ;-)
>
> What about: "(Note: changes in files already known to git, and even new
> changes done after a previous `git add` for a given file, still must
> be staged again)"
Or maybe: "by using gitlink:git-add[1] to add new content (of either new
or newly modified files) to the next commit."
Man pages are reference documentation, so I figure it's OK to sacrifice
a little newbie-friendliness for accuracy and concision.
I dunno, I think basically the right content is there and Junio should
just commit it (after the git-rm change?), and then allow the rest of us
nitpickers to submit patches against the result to our heart's
content....
^ permalink raw reply
* Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
From: H. Peter Anvin @ 2006-12-10 21:01 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jakub Narebski, Jeff Garzik, Martin Langhoff, Git Mailing List,
Rogan Dawes, Kernel Org Admin
In-Reply-To: <Pine.LNX.4.64.0612101129190.12500@woody.osdl.org>
Linus Torvalds wrote:
>
> Now, hopefully some of it will be in the disk cache, but when the
> mirroring happens, it will basically blow the disk caches away totally
> (when using the "--checksum" option), and then you literally have tens of
> seconds to generate that one top-level page.
>
If that was the only time that happened, it would be a non-issue, since
that only happens once every 96 hours. However, the problem is that we
now have lots of large datasets that blow out the caches on a much more
frequent basis.
^ permalink raw reply
* Re: globs in partial checkout?
From: Michael S. Tsirkin @ 2006-12-10 21:07 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List, Junio C Hamano
In-Reply-To: <Pine.LNX.4.64.0612101209370.12500@woody.osdl.org>
> However, since about 1.4.4.1 or so, it now does the same filename
> expansion as "git add" etc does, which means that you can give it a
> directory name and it will check out everything under that directory, or
> you can give it a pattern, and it should glob it. But it sounds like you
> may have a slightly older version of git (the pathname matching really is
> fairly recent).
Seems like this was post-1.4.4.1. Just updated to 1.4.4.2 and this works for me now,
thanks.
--
^ permalink raw reply
* Re: [PATCH 2/3] merge-recursive: make empty tree a known object
From: Junio C Hamano @ 2006-12-10 21:21 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Johannes Schindelin, Shawn O. Pearce, git
In-Reply-To: <Pine.LNX.4.64.0612101030390.12500@woody.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> So you could make "read_sha1_file()" just have a special case for known
> objects at the end. If the pack entry fails, the loose file case fails,
> then rather than returning NULL at the end, you could have a list of known
> fixed objects..
That is fine by me. We would benefit from an empty blob and an
empty tree.
^ permalink raw reply
* Re: [PATCH] Make cvsexportcommit work with filenames with spaces and non-ascii characters.
From: Junio C Hamano @ 2006-12-10 21:27 UTC (permalink / raw)
To: Robin Rosenberg; +Cc: git
In-Reply-To: <200612101639.22397.robin.rosenberg@dewire.com>
Robin Rosenberg <robin.rosenberg@dewire.com> writes:
>> Well, have you tried this?
>>
>> git diff-tree -p --binary fe142b3a | git apply --summary --numstat
> of course not. I didn't understand it. Why can't it tell me about removed
> binary files, so I could remove the git-diff-tree invocation to find out about
> added/removed files?
Maybe I am missing something. It tells you about added or
removed files (either binary or non-binary).
I'd prepare a pair of practice patch files. "forward" has
creation, "reverse" has addition.
$ git diff-tree -p --binary fe142b3a >forward.patch
$ git diff-tree -R -p --binary fe142b3a >reverse.patch
$ git apply --summary --numstat forward.patch
50 20 git-cvsexportcommit.perl
145 0 t/t9200-git-cvsexportcommit.sh
- - t/test9200a.png
- - t/test9200b.png
create mode 100755 t/t9200-git-cvsexportcommit.sh
create mode 100644 t/test9200a.png
create mode 100644 t/test9200b.png
$ git apply --summary --numstat reverse.patch
20 50 git-cvsexportcommit.perl
0 145 t/t9200-git-cvsexportcommit.sh
- - t/test9200a.png
- - t/test9200b.png
delete mode 100755 t/t9200-git-cvsexportcommit.sh
delete mode 100644 t/test9200a.png
delete mode 100644 t/test9200b.png
> This fuzz gives some, but not much slack. Reverting the option could be a
> good idea.
>
> Update follows.
Ok.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox