* Re: Performance impact of a large number of commits
From: david @ 2008-10-24 20:11 UTC (permalink / raw)
To: Samuel Abels; +Cc: git
In-Reply-To: <1224878202.7566.25.camel@localhost>
On Fri, 24 Oct 2008, Samuel Abels wrote:
> On Fri, 2008-10-24 at 12:43 -0700, david@lang.hm wrote:
>> 100,000 commits per day??
>>
>> that's 1.5 commits/second. what is updating files that quickly?
>
> This is an automated process taking snapshots of rapidly changing files
> containing statistical data. Unfortunately, our needs go beyond what a
> versioning file system has to offer, and the data is an unstructured
> text file (in other words, using a relational database is not a good
> option).
>
>> I suspect that you will have some issues here, but it's going to depend on
>> how many files get updated each 3/4 of a second.
>
> That would be 5 to 10 changed files per commit, and those are passed to
> git commit explicitly (i.e., walking the tree to stat files for finding
> changes is not necessary).
I suspect that your limits would be filesystem/OS limits more than git
limits
at 5-10 files/commit you are going to be creating .5-1m files/day, even
spread across 256 directories this is going to be a _lot_ of files.
packing this may help (depending on how much the files change), but with
this many files the work of doing the packing would be expensive.
David Lang
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Giuseppe Bilotta @ 2008-10-24 20:06 UTC (permalink / raw)
To: david; +Cc: Junio C Hamano, Jakub Narebski, Jean-Luc Herren, git
In-Reply-To: <alpine.DEB.1.10.0810241251490.27333@asgard.lang.hm>
On Fri, Oct 24, 2008 at 9:53 PM, <david@lang.hm> wrote:
> I just had what's probably a silly thought.
>
> how close is a zit setup to a subproject setup?
Honestly, I haven't the slightest idea how they work. My
understanding, which could be completely wrong, is that they are
full-fledged git repositories, and that additional metadata at the top
level takes care of understanding what ref is needed for each toplevel
project. If this is true, using them wouldn't simplify zit, but rather
make it more complex (and space intensive).
--
Giuseppe "Oblomov" Bilotta
^ permalink raw reply
* Re: Performance impact of a large number of commits
From: Samuel Abels @ 2008-10-24 19:56 UTC (permalink / raw)
To: david; +Cc: git
In-Reply-To: <alpine.DEB.1.10.0810241240100.27333@asgard.lang.hm>
On Fri, 2008-10-24 at 12:43 -0700, david@lang.hm wrote:
> 100,000 commits per day??
>
> that's 1.5 commits/second. what is updating files that quickly?
This is an automated process taking snapshots of rapidly changing files
containing statistical data. Unfortunately, our needs go beyond what a
versioning file system has to offer, and the data is an unstructured
text file (in other words, using a relational database is not a good
option).
> I suspect that you will have some issues here, but it's going to depend on
> how many files get updated each 3/4 of a second.
That would be 5 to 10 changed files per commit, and those are passed to
git commit explicitly (i.e., walking the tree to stat files for finding
changes is not necessary).
-Samuel
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: david @ 2008-10-24 19:54 UTC (permalink / raw)
To: Giuseppe Bilotta; +Cc: Junio C Hamano, Jakub Narebski, Jean-Luc Herren, git
In-Reply-To: <cb7bb73a0810241251w4c2486a0x4684a25b364ebbbb@mail.gmail.com>
On Fri, 24 Oct 2008, Giuseppe Bilotta wrote:
> On Fri, Oct 24, 2008 at 9:46 PM, <david@lang.hm> wrote:
>> On Fri, 24 Oct 2008, Giuseppe Bilotta wrote:
>>>
>>> It seems that tags, hooks, branches and description can be done with.
>>
>> do you mean 'can be done away with'?
>
> Ahem. Yes. I've got a patch ready for zit that gets rid of them.
>
> (A smarter way would be to create a template, but I'm not smart.)
>
>>> info contains exclude which is rather essential,
>>
>> is it? by default everything in this file is commented out. And with you
>> only adding files explicitly why would it ever need to excluded anything?
>
> Zit does
> echo "*" > $GIT_DIR/info/exclude
> and yes it sucks to use a whole block for a file that only contains
> one character. Suggestions welcome.
can this be configured in the config file?
> The reason why we want the exclude is that when you do zit status
> somefile you don't want every other file in the directory to come up
> as 'not tracked'.
good point.
David Lang
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: david @ 2008-10-24 19:53 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jakub Narebski, Giuseppe Bilotta, Jean-Luc Herren, git
In-Reply-To: <7vabct3l1e.fsf@gitster.siamese.dyndns.org>
On Fri, 24 Oct 2008, Junio C Hamano wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>
>> "Giuseppe Bilotta" <giuseppe.bilotta@gmail.com> writes:
>>> On Fri, Oct 24, 2008 at 1:23 AM, Jean-Luc Herren <jlh@gmx.ch> wrote:
>>
>>>> If you decide against a shared repository, maybe you want to
>>>> consider to not use ".zit.file/", but ".zit/file/" as the
>>>> repository? This would reduce the clutter to a single directory,
>>>> just like with ".git". And moving files around wouldn't be that
>>>> much complicated.
>>>
>>> Right. I'll give that a shot.
>>
>> By the way RCS which I use for version control of single files use
>> both approaches: it can store 'file,v' alongside 'file' (just like
>> your '.zit.file/' or '.file.git/'), but it can also store files on
>> per-directory basis in 'RCS/' subdirectory (proposed '.zit/file/' or
>> '.zit/file.git/' solution)
>
> I am not opposed to the wish to track a single file (but I have to say I
> am not personally in need for such a feature), but I have to wonder from
> the technical point of view if one-repo-per-file is the right approach.
I just had what's probably a silly thought.
how close is a zit setup to a subproject setup?
David Lang
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Giuseppe Bilotta @ 2008-10-24 19:51 UTC (permalink / raw)
To: david; +Cc: Junio C Hamano, Jakub Narebski, Jean-Luc Herren, git
In-Reply-To: <alpine.DEB.1.10.0810241244170.27333@asgard.lang.hm>
On Fri, Oct 24, 2008 at 9:46 PM, <david@lang.hm> wrote:
> On Fri, 24 Oct 2008, Giuseppe Bilotta wrote:
>>
>> It seems that tags, hooks, branches and description can be done with.
>
> do you mean 'can be done away with'?
Ahem. Yes. I've got a patch ready for zit that gets rid of them.
(A smarter way would be to create a template, but I'm not smart.)
>> info contains exclude which is rather essential,
>
> is it? by default everything in this file is commented out. And with you
> only adding files explicitly why would it ever need to excluded anything?
Zit does
echo "*" > $GIT_DIR/info/exclude
and yes it sucks to use a whole block for a file that only contains
one character. Suggestions welcome.
The reason why we want the exclude is that when you do zit status
somefile you don't want every other file in the directory to come up
as 'not tracked'.
--
Giuseppe "Oblomov" Bilotta
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: david @ 2008-10-24 19:46 UTC (permalink / raw)
To: Giuseppe Bilotta; +Cc: Junio C Hamano, Jakub Narebski, Jean-Luc Herren, git
In-Reply-To: <cb7bb73a0810241242y7467f6fexcca4b7cd768e7992@mail.gmail.com>
On Fri, 24 Oct 2008, Giuseppe Bilotta wrote:
> I was slowly writing a reply but it seems David beat me to it, so here
> goes a couple of additional comments.
>
> On Fri, Oct 24, 2008 at 9:11 PM, <david@lang.hm> wrote:
>> On Fri, 24 Oct 2008, Junio C Hamano wrote:
>>> Running "git init" in an empty directory consumes about 100k of diskspace
>>> on the machine I am typing this on, and you should be able to share most
>>> of them (except one 41-byte file that is the branch tip ref) when you
>>> track many files inside a single directory by using a single repository,
>>> one branch per file (or "one set of branches per file") model.
>>
>> the reason to use seperate repos is to ease the work involved if you need to
>> move that file (and it's repo) elsewhere.
>
> Precisely. The one-repo-per-file is just the simplest and most
> flexible solution. But yes, I have to admit I hadn't looked into disk
> usage, and indeed we should try and squeeze this as much as possible.
>
>> with the git directory being under .zit, would it be possible to link the
>> things that are nessasary togeather?
>
> I'm not sure about _which_ files could be shared.
>
>> hmm, looking at this in more detail.
>>
>> about 44K of diskspace is used by the .sample hook files, so those can be
>> removed
>
> Exactly. I'm setting up zit to prepare its repos to a more compact
> form, and getting rid of hooks and description is the first step.
>
>> the remaining 56K is mostly directories eating up a disk block
>>
>> find . -ls
>> 200367 4 drwxr-xr-x 7 dlang users 4096 Oct 24 12:00 .
>> 200368 4 drwxr-xr-x 4 dlang users 4096 Oct 24 12:00 ./refs
>> 200369 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
>> ./refs/heads
>> 200370 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
>> ./refs/tags
>> 200371 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
>> ./branches
>> 200372 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./hooks
>> 200373 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./info
>> 1798469 4 -rw-r--r-- 1 dlang users 240 Oct 24 12:00
>> ./info/exclude
>> 1600716 4 -rw-r--r-- 1 dlang users 58 Oct 24 12:00
>> ./description
>> 200374 4 drwxr-xr-x 4 dlang users 4096 Oct 24 12:00 ./objects
>> 200375 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
>> ./objects/pack
>> 200376 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
>> ./objects/info
>> 1600717 4 -rw-r--r-- 1 dlang users 23 Oct 24 12:00 ./HEAD
>> 1600719 4 -rw-r--r-- 1 dlang users 92 Oct 24 12:00 ./config
>>
>> how many of these are _really_ nessasary?
>
> For starters, I'm wondering if setting core.preferSymlinkRefs would be
> useful here. Does it break sometihng?
>
>> tags, info, hooks, branches, and description could probably be skipped for
>> the common zit case, as long as they can be created as needed.
>
> It seems that tags, hooks, branches and description can be done with.
do you mean 'can be done away with'?
> info contains exclude which is rather essential,
is it? by default everything in this file is commented out. And with you
only adding files explicitly why would it ever need to excluded anything?
David Lang
^ permalink raw reply
* Re: Performance impact of a large number of commits
From: david @ 2008-10-24 19:43 UTC (permalink / raw)
To: Samuel Abels; +Cc: git
In-Reply-To: <1224874946.7566.13.camel@localhost>
On Fri, 24 Oct 2008, Samuel Abels wrote:
> Hi,
>
> I am considering Git to maintain a repository of approximately 300.000
> files totaling 1 GB, with a number of ~100.000 commits per day, all in
> one single branch. The only operations performed are "git commit", "git
> show", and "git checkout", and all on one local machine. Does this sound
> like a reasonable thing to do with Git?
100,000 commits per day??
that's 1.5 commits/second. what is updating files that quickly?
I suspect that you will have some issues here, but it's going to depend on
how many files get updated each 3/4 of a second.
David Lang
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Giuseppe Bilotta @ 2008-10-24 19:42 UTC (permalink / raw)
To: david; +Cc: Junio C Hamano, Jakub Narebski, Jean-Luc Herren, git
In-Reply-To: <alpine.DEB.1.10.0810241159290.27333@asgard.lang.hm>
I was slowly writing a reply but it seems David beat me to it, so here
goes a couple of additional comments.
On Fri, Oct 24, 2008 at 9:11 PM, <david@lang.hm> wrote:
> On Fri, 24 Oct 2008, Junio C Hamano wrote:
>> Running "git init" in an empty directory consumes about 100k of diskspace
>> on the machine I am typing this on, and you should be able to share most
>> of them (except one 41-byte file that is the branch tip ref) when you
>> track many files inside a single directory by using a single repository,
>> one branch per file (or "one set of branches per file") model.
>
> the reason to use seperate repos is to ease the work involved if you need to
> move that file (and it's repo) elsewhere.
Precisely. The one-repo-per-file is just the simplest and most
flexible solution. But yes, I have to admit I hadn't looked into disk
usage, and indeed we should try and squeeze this as much as possible.
> with the git directory being under .zit, would it be possible to link the
> things that are nessasary togeather?
I'm not sure about _which_ files could be shared.
> hmm, looking at this in more detail.
>
> about 44K of diskspace is used by the .sample hook files, so those can be
> removed
Exactly. I'm setting up zit to prepare its repos to a more compact
form, and getting rid of hooks and description is the first step.
> the remaining 56K is mostly directories eating up a disk block
>
> find . -ls
> 200367 4 drwxr-xr-x 7 dlang users 4096 Oct 24 12:00 .
> 200368 4 drwxr-xr-x 4 dlang users 4096 Oct 24 12:00 ./refs
> 200369 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
> ./refs/heads
> 200370 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
> ./refs/tags
> 200371 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
> ./branches
> 200372 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./hooks
> 200373 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./info
> 1798469 4 -rw-r--r-- 1 dlang users 240 Oct 24 12:00
> ./info/exclude
> 1600716 4 -rw-r--r-- 1 dlang users 58 Oct 24 12:00
> ./description
> 200374 4 drwxr-xr-x 4 dlang users 4096 Oct 24 12:00 ./objects
> 200375 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
> ./objects/pack
> 200376 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00
> ./objects/info
> 1600717 4 -rw-r--r-- 1 dlang users 23 Oct 24 12:00 ./HEAD
> 1600719 4 -rw-r--r-- 1 dlang users 92 Oct 24 12:00 ./config
>
> how many of these are _really_ nessasary?
For starters, I'm wondering if setting core.preferSymlinkRefs would be
useful here. Does it break sometihng?
> tags, info, hooks, branches, and description could probably be skipped for
> the common zit case, as long as they can be created as needed.
It seems that tags, hooks, branches and description can be done with.
info contains exclude which is rather essential, and this is something
that could be shared across repositories. Also, we could spare a block
by removing info, moving exclude to the .git dir and setting
core.excludesfile appropriately
> the objects directory will eat up more space as revisions are checked in
> (and more sub-directories are created), would it make sense to have a config
> option to do a flat objects directory instead of the current fan-out?
This is probably the biggest remaining spacewaste. Typical zit usage
will generate a rather small number of objects, so flattening the
object store for the repo wouldn't be a bad idea. Is that possible?
--
Giuseppe "Oblomov" Bilotta
^ permalink raw reply
* Performance impact of a large number of commits
From: Samuel Abels @ 2008-10-24 19:02 UTC (permalink / raw)
To: git
Hi,
I am considering Git to maintain a repository of approximately 300.000
files totaling 1 GB, with a number of ~100.000 commits per day, all in
one single branch. The only operations performed are "git commit", "git
show", and "git checkout", and all on one local machine. Does this sound
like a reasonable thing to do with Git?
-Samuel
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: david @ 2008-10-24 19:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jakub Narebski, Giuseppe Bilotta, Jean-Luc Herren, git
In-Reply-To: <7vabct3l1e.fsf@gitster.siamese.dyndns.org>
On Fri, 24 Oct 2008, Junio C Hamano wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>
>> "Giuseppe Bilotta" <giuseppe.bilotta@gmail.com> writes:
>>> On Fri, Oct 24, 2008 at 1:23 AM, Jean-Luc Herren <jlh@gmx.ch> wrote:
>>
>>>> If you decide against a shared repository, maybe you want to
>>>> consider to not use ".zit.file/", but ".zit/file/" as the
>>>> repository? This would reduce the clutter to a single directory,
>>>> just like with ".git". And moving files around wouldn't be that
>>>> much complicated.
>>>
>>> Right. I'll give that a shot.
>>
>> By the way RCS which I use for version control of single files use
>> both approaches: it can store 'file,v' alongside 'file' (just like
>> your '.zit.file/' or '.file.git/'), but it can also store files on
>> per-directory basis in 'RCS/' subdirectory (proposed '.zit/file/' or
>> '.zit/file.git/' solution)
>
> I am not opposed to the wish to track a single file (but I have to say I
> am not personally in need for such a feature), but I have to wonder from
> the technical point of view if one-repo-per-file is the right approach.
>
> Running "git init" in an empty directory consumes about 100k of diskspace
> on the machine I am typing this on, and you should be able to share most
> of them (except one 41-byte file that is the branch tip ref) when you
> track many files inside a single directory by using a single repository,
> one branch per file (or "one set of branches per file") model.
the reason to use seperate repos is to ease the work involved if you need
to move that file (and it's repo) elsewhere.
with the git directory being under .zit, would it be possible to link the
things that are nessasary togeather?
hmm, looking at this in more detail.
about 44K of diskspace is used by the .sample hook files, so those can be
removed
the remaining 56K is mostly directories eating up a disk block
find . -ls
200367 4 drwxr-xr-x 7 dlang users 4096 Oct 24 12:00 .
200368 4 drwxr-xr-x 4 dlang users 4096 Oct 24 12:00 ./refs
200369 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./refs/heads
200370 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./refs/tags
200371 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./branches
200372 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./hooks
200373 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./info
1798469 4 -rw-r--r-- 1 dlang users 240 Oct 24 12:00 ./info/exclude
1600716 4 -rw-r--r-- 1 dlang users 58 Oct 24 12:00 ./description
200374 4 drwxr-xr-x 4 dlang users 4096 Oct 24 12:00 ./objects
200375 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./objects/pack
200376 4 drwxr-xr-x 2 dlang users 4096 Oct 24 12:00 ./objects/info
1600717 4 -rw-r--r-- 1 dlang users 23 Oct 24 12:00 ./HEAD
1600719 4 -rw-r--r-- 1 dlang users 92 Oct 24 12:00 ./config
how many of these are _really_ nessasary?
tags, info, hooks, branches, and description could probably be skipped for
the common zit case, as long as they can be created as needed.
If git has problems with these not existing, would it make sense to make
git survive if they are missing and create them if needed?
the objects directory will eat up more space as revisions are checked in
(and more sub-directories are created), would it make sense to have a
config option to do a flat objects directory instead of the current
fan-out?
the other option with objects would be to look into having a common
objects fan-out directory, but have the pack directory be per file. This
would allow you to seperate out one files stuff by creating packs for it
and then grabbing everything in the per-file directory.
thoughts?
David Lang
^ permalink raw reply
* Re: git performance
From: Jakub Narebski @ 2008-10-24 19:06 UTC (permalink / raw)
To: George Shammas; +Cc: git
In-Reply-To: <dfdaadcd0810241042k1469fc30x62daa19273404edc@mail.gmail.com>
"George Shammas" <georgyo@gmail.com> writes:
> If you are really trying to backup a filesystem, you may want to look
> at a filesystem that can do snapshots, it would be a lot more
> efficient then a version control system. Such as NILFS and ZFS.
>
> http://en.wikipedia.org/wiki/NILFS
> http://en.wikipedia.org/wiki/ZFS
Or ext3cow, or (currently in early stages of development) Tux3
http://en.wikipedia.org/wiki/Ext3cow
http://en.wikipedia.org/wiki/Tux3
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Junio C Hamano @ 2008-10-24 18:28 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Giuseppe Bilotta, Jean-Luc Herren, git
In-Reply-To: <m38wsei8ne.fsf@localhost.localdomain>
Jakub Narebski <jnareb@gmail.com> writes:
> "Giuseppe Bilotta" <giuseppe.bilotta@gmail.com> writes:
>> On Fri, Oct 24, 2008 at 1:23 AM, Jean-Luc Herren <jlh@gmx.ch> wrote:
>
>> > If you decide against a shared repository, maybe you want to
>> > consider to not use ".zit.file/", but ".zit/file/" as the
>> > repository? This would reduce the clutter to a single directory,
>> > just like with ".git". And moving files around wouldn't be that
>> > much complicated.
>>
>> Right. I'll give that a shot.
>
> By the way RCS which I use for version control of single files use
> both approaches: it can store 'file,v' alongside 'file' (just like
> your '.zit.file/' or '.file.git/'), but it can also store files on
> per-directory basis in 'RCS/' subdirectory (proposed '.zit/file/' or
> '.zit/file.git/' solution)
I am not opposed to the wish to track a single file (but I have to say I
am not personally in need for such a feature), but I have to wonder from
the technical point of view if one-repo-per-file is the right approach.
Running "git init" in an empty directory consumes about 100k of diskspace
on the machine I am typing this on, and you should be able to share most
of them (except one 41-byte file that is the branch tip ref) when you
track many files inside a single directory by using a single repository,
one branch per file (or "one set of branches per file") model.
^ permalink raw reply
* Re: git performance
From: Jeff King @ 2008-10-24 18:20 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Edward Ned Harvey, git
In-Reply-To: <alpine.LFD.2.00.0810241050310.3287@nehalem.linux-foundation.org>
On Fri, Oct 24, 2008 at 10:53:20AM -0700, Linus Torvalds wrote:
> > echo 1 >/proc/sys/vm/drop_caches
>
> Use "echo 3" instead of "1".
>
> It's actually a bitmask, with bit 0 being "data" (pagecache) and bit 1
> being "metadata" (inodes and directory caches).
>
> And since git (or any SCM) is very metadata-intensive, you really should
> make sure to drop metadata too, otherwise your caches won't be really very
> cold at all.
>
> (But it obviously depends on the operation you're testing - some are more
> about the inodes and directories, others are about file data access).
Ah, thanks. In this case, he was interested in walking the directory
tree, so the metadata caching was indeed very important.
-Peff
^ permalink raw reply
* Re: git performance
From: Linus Torvalds @ 2008-10-24 17:53 UTC (permalink / raw)
To: Jeff King; +Cc: Edward Ned Harvey, git
In-Reply-To: <20081024142947.GB11568@coredump.intra.peff.net>
On Fri, 24 Oct 2008, Jeff King wrote:
>
> Side note: on Linux, it is much easier to clear the cache via
>
> echo 1 >/proc/sys/vm/drop_caches
Use "echo 3" instead of "1".
It's actually a bitmask, with bit 0 being "data" (pagecache) and bit 1
being "metadata" (inodes and directory caches).
And since git (or any SCM) is very metadata-intensive, you really should
make sure to drop metadata too, otherwise your caches won't be really very
cold at all.
(But it obviously depends on the operation you're testing - some are more
about the inodes and directories, others are about file data access).
Linus
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Giuseppe Bilotta @ 2008-10-24 17:48 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Johannes Sixt, git
In-Reply-To: <alpine.DEB.1.00.0810241943470.22125@pacific.mpi-cbg.de.mpi-cbg.de>
On Fri, Oct 24, 2008 at 7:44 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Thu, 23 Oct 2008, Johannes Sixt wrote:
>
>> Giuseppe Bilotta schrieb:
>> > Zit will create a directory .zit.file to hold a git repository
>> > tracking the single file .zit.file/file, which is just a hard link to
>> > file.
>>
>> git breaks hard links, mind you! (Just in case you check out older
>> versions and you wonder why your "real" file is not updated).
>>
>> But there's a recent patch by Dscho floating around that takes care of
>> the hard link case.
>
> Yep, I still want to work on it; it breaks on one of Junio's machines.
Well, it's not needed by Zit anymore, but there was someone else
asking about on the ml recently, too 8-)
--
Giuseppe "Oblomov" Bilotta
^ permalink raw reply
* Re: git performance
From: George Shammas @ 2008-10-24 17:42 UTC (permalink / raw)
To: git
In-Reply-To: <20081024142947.GB11568@coredump.intra.peff.net>
If you are really trying to backup a filesystem, you may want to look
at a filesystem that can do snapshots, it would be a lot more
efficient then a version control system. Such as NILFS and ZFS.
http://en.wikipedia.org/wiki/NILFS
http://en.wikipedia.org/wiki/ZFS
Both these will allow you to look at changed files over time. NILFS is
slightlly diffrent in that it doesn't take snapshots, because it never
deletes, so you can rollback every change on a file. They both also
allow each user to rollback their own files if they wanted to, so if
this is your goal, source code version control is not for you, and a
good file system is for you.
-G
On Fri, Oct 24, 2008 at 10:29 AM, Jeff King <peff@peff.net> wrote:
> On Fri, Oct 24, 2008 at 12:15:19AM -0400, Edward Ned Harvey wrote:
>
>> Feel free to forward to the list, if anyone's still talking about it.
>> I already un-subscribed.
>
> Posting is not limited to subscribers, so you can happily continue the
> conversation there by cc'ing the list (and I am cc'ing the list here).
>
>> I did my benchmarking at least two months ago, so I forgot the exact
>> results now, so I ran the benchmark once just now. I also downloaded
>> git, and did "git status" for comparison. I rebooted the system in
>> between each trial run, to clear the cache. Here's the results:
>
> Side note: on Linux, it is much easier to clear the cache via
>
> echo 1 >/proc/sys/vm/drop_caches
>
> than to reboot for each benchmark.
>
>> Local disk mirror "time git status" on the same tree. 17,468 versioned files, so the whole tree is 30,647 including .git files
>> 0m 25s cold cache
>> 0m 0.2s warm cache trial 1
>> 0m 0.2s warm cache trial 2
>
> Hmm. That's a lot of increase in files for .git. Did you try repacking
> and then running your test?
>
>> I questioned whether svn and git were causing unnecessary overhead.
>
> Sure, they are doing more than just walking. So there is overhead, but
> it's hard to say how much is unnecessary. However, if you were working
> with an unpacked git, then it may have had to open() a lot of files in
> the object db (keep in mind that status doesn't just show the difference
> between the working tree and the index; it shows the difference between
> the index and the last commit. So maybe "git diff" would be a more
> accurate comparison).
>
>> Conclusions:
>> * For "status" operations on cold cache, large file count, Neither the
>> performance of git or svn approaches the ideal. Both are an order of
>> magnitude slower than ideal, which is still assuming "ideal" requires
>> walking the tree. A better ideal avoids the need to walk the tree,
>> and has near-zero total cost.
>
> Try your git benchmark again with a packed repo, and I think you will
> find it approaches the time it takes to walk the tree.
>
> That being said, if walking the tree is unacceptable to you, then no,
> current git won't work. You would need to patch it to use inotify (once
> upon a time there was some discussion of this, but it never went
> anywhere -- I guess most people work on machines where they can keep the
> cache relatively warm).
>
> -Peff
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Johannes Schindelin @ 2008-10-24 17:44 UTC (permalink / raw)
To: Johannes Sixt; +Cc: Giuseppe Bilotta, git
In-Reply-To: <49007623.1060606@viscovery.net>
Hi,
On Thu, 23 Oct 2008, Johannes Sixt wrote:
> Giuseppe Bilotta schrieb:
> > Zit will create a directory .zit.file to hold a git repository
> > tracking the single file .zit.file/file, which is just a hard link to
> > file.
>
> git breaks hard links, mind you! (Just in case you check out older
> versions and you wonder why your "real" file is not updated).
>
> But there's a recent patch by Dscho floating around that takes care of
> the hard link case.
Yep, I still want to work on it; it breaks on one of Junio's machines.
Ciao,
Dscho
^ permalink raw reply
* Re: [RFC PATCH v2] fetch-pack: log(n)-transmission find_common()
From: Nicolas Pitre @ 2008-10-24 15:49 UTC (permalink / raw)
To: Thomas Rast; +Cc: git
In-Reply-To: <1224790716-20551-1-git-send-email-trast@student.ethz.ch>
On Thu, 23 Oct 2008, Thomas Rast wrote:
> Replaces the existing simple history search with a more sophisticated
> algorithm:
>
> 1) Walk history with exponentially increasing stride lengths; i.e.,
> send the 1st commit, then the 2nd after that, then the 4th after
> that, and so on.
>
> 2) Bisect the resulting intervals.
>
> Combined with tracking the "outstanding haves" so that we can detect
> which sha1s were never ACKed by upload-pack (and hence not common),
> this gives O(log(n)) required "have" lines.
>
> Unfortunately this cannot work if the server sends fake ACKs, so we
> introduce a capability 'exp-stride' which instructs upload-pack to
> disable ok_to_give_up(). (Which incidentally saves the server a lot
> of work.)
>
> Signed-off-by: Thomas Rast <trast@student.ethz.ch>
>
> ---
>
> This is a simple resend of v2, in the hope of attracting some
> discussion or at least encouraging words this time around.
OK, I gave this a quick try, and fetch operations appear to make their
mind about what to do quicker. Some fetch requests which used to take
up to 5 seconds are somewhat faster. I have no formal measurements
though.
Nicolas
^ permalink raw reply
* Re: [PATCH] git-svn: change dashed git-config to git config
From: Deskin Miller @ 2008-10-24 15:27 UTC (permalink / raw)
To: gitster; +Cc: git, Eric Wong
In-Reply-To: <20081024091544.GB27362@untitled>
On Fri, Oct 24, 2008 at 02:15:44AM -0700, Eric Wong wrote:
> Deskin Miller <deskinm@umich.edu> wrote:
> > ---
> > When testing git-svn changes, I'll often use git-svn.perl directly on a
> > repository of mine, to see how things work. Recently I had this happen:
> >
> > $ git-svn.perl info
> > Use of uninitialized value in scalar chomp at ../git/git-svn.perl line 1202.
> > Use of uninitialized value in concatenation (.) or string at ../git/git-svn.perl line 1203.
> >
> > but the expected output showed up correctly, giving the SVN-like info.
> >
> > git svn info appeared to work just fine though, despite there being essentially
> > no difference between the two. However, if I do the following, this is what I
> > see:
> >
> > $ env /usr/libexec/git-core/git-svn info
> > Can't exec "git-config": No such file or directory at /usr/libexec/git-core/git-svn line 1202.
> > Use of uninitialized value in scalar chomp at /usr/libexec/git-core/git-svn line 1202.
> >
> > Again, the SVN info would still appear.
> >
> > Regardless, I think we want to change this call to use the non-dashed form.
>
> Thanks Deskin,
>
> Acked-by: Eric Wong <normalperson@yhbt.net>
Sorry, patch should have had a
Signed-off-by: Deskin Miller <deskinm@umich.edu>
Feel free to forge it.
> > Deskin Miller
> >
> > git-svn.perl | 2 +-
> > 1 files changed, 1 insertions(+), 1 deletions(-)
> >
> > diff --git a/git-svn.perl b/git-svn.perl
> > index 33e1b50..2e68c68 100755
> > --- a/git-svn.perl
> > +++ b/git-svn.perl
> > @@ -1126,7 +1126,7 @@ sub read_repo_config {
> > my $v = $opts->{$o};
> > my ($key) = ($o =~ /^([a-zA-Z\-]+)/);
> > $key =~ s/-//g;
> > - my $arg = 'git-config';
> > + my $arg = 'git config';
> > $arg .= ' --int' if ($o =~ /[:=]i$/);
> > $arg .= ' --bool' if ($o !~ /[:=][sfi]$/);
> > if (ref $v eq 'ARRAY') {
> > --
> > 1.6.0.2.554.g3041b
^ permalink raw reply
* Re: git performance
From: Jeff King @ 2008-10-24 14:29 UTC (permalink / raw)
To: Edward Ned Harvey; +Cc: git
In-Reply-To: <000001c9358f$232bac70$69830550$@com>
On Fri, Oct 24, 2008 at 12:15:19AM -0400, Edward Ned Harvey wrote:
> Feel free to forward to the list, if anyone's still talking about it.
> I already un-subscribed.
Posting is not limited to subscribers, so you can happily continue the
conversation there by cc'ing the list (and I am cc'ing the list here).
> I did my benchmarking at least two months ago, so I forgot the exact
> results now, so I ran the benchmark once just now. I also downloaded
> git, and did "git status" for comparison. I rebooted the system in
> between each trial run, to clear the cache. Here's the results:
Side note: on Linux, it is much easier to clear the cache via
echo 1 >/proc/sys/vm/drop_caches
than to reboot for each benchmark.
> Local disk mirror "time git status" on the same tree. 17,468 versioned files, so the whole tree is 30,647 including .git files
> 0m 25s cold cache
> 0m 0.2s warm cache trial 1
> 0m 0.2s warm cache trial 2
Hmm. That's a lot of increase in files for .git. Did you try repacking
and then running your test?
> I questioned whether svn and git were causing unnecessary overhead.
Sure, they are doing more than just walking. So there is overhead, but
it's hard to say how much is unnecessary. However, if you were working
with an unpacked git, then it may have had to open() a lot of files in
the object db (keep in mind that status doesn't just show the difference
between the working tree and the index; it shows the difference between
the index and the last commit. So maybe "git diff" would be a more
accurate comparison).
> Conclusions:
> * For "status" operations on cold cache, large file count, Neither the
> performance of git or svn approaches the ideal. Both are an order of
> magnitude slower than ideal, which is still assuming "ideal" requires
> walking the tree. A better ideal avoids the need to walk the tree,
> and has near-zero total cost.
Try your git benchmark again with a packed repo, and I think you will
find it approaches the time it takes to walk the tree.
That being said, if walking the tree is unacceptable to you, then no,
current git won't work. You would need to patch it to use inotify (once
upon a time there was some discussion of this, but it never went
anywhere -- I guess most people work on machines where they can keep the
cache relatively warm).
-Peff
^ permalink raw reply
* Re: [PATCH 3/5] refactor userdiff textconv code
From: Jeff King @ 2008-10-24 14:08 UTC (permalink / raw)
To: Johannes Sixt; +Cc: git, Junio C Hamano, Matthieu Moy
In-Reply-To: <4901D557.90106@viscovery.net>
On Fri, Oct 24, 2008 at 04:01:59PM +0200, Johannes Sixt wrote:
> Your reasoning makes sense, of course. (I thought that forgoing textconv
> in these cases would have a simpler implementation; but it can't be a lot
> simpler than yours.)
Yeah, I think preventing it entirely for this case would be much more
complex, since you would have to realize much sooner that a textconv was
going to happen.
> > +test_expect_failure 'textconv does not act on symlinks' '
> Can we trust your solution if you still have 'test_expect_failure' here? ;-)
Heh. It helps if I commit before format-patch. ;) Fortunately, this was
the only change I missed out on sending.
-Peff
^ permalink raw reply
* Re: [PATCH 3/5] refactor userdiff textconv code
From: Johannes Sixt @ 2008-10-24 14:01 UTC (permalink / raw)
To: Jeff King; +Cc: git, Junio C Hamano, Matthieu Moy
In-Reply-To: <20081024135132.GA11568@coredump.intra.peff.net>
Jeff King schrieb:
> On Fri, Oct 24, 2008 at 09:15:54AM +0200, Johannes Sixt wrote:
> Actually, I don't think this is right. The typechange has been broken
> into two parts: the removal of the file contents and the addition of the
> symlink. So the first part _should_ use the textconv, since it is just
> comparing the file contents to /dev/null. But the second part should
> not, since it is by definition just the text of the symlink. Ditto for
> gitlinks, which have a special text representation. So how about this?
Your reasoning makes sense, of course. (I thought that forgoing textconv
in these cases would have a simpler implementation; but it can't be a lot
simpler than yours.)
> diff --git a/diff.c b/diff.c
> index 52feba7..ce1317d 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -289,6 +289,9 @@ static int diff_filespec_textconv(struct diff_filespec *one)
> if (!DIFF_FILE_VALID(one))
> return 0;
>
> + if (!S_ISREG(one->mode))
> + return 0;
> +
> diff_filespec_load_driver(one);
> if (!one->driver->textconv)
> return 0;
...
> +test_expect_failure 'textconv does not act on symlinks' '
Can we trust your solution if you still have 'test_expect_failure' here? ;-)
-- Hannes
^ permalink raw reply
* [PATCH/RFC] Introduce a built-in attribute "text"
From: Michael J Gruber @ 2008-10-24 13:55 UTC (permalink / raw)
To: git; +Cc: Michael J Gruber
"text is the opposite of "binary": It sets the attributes "crlf" and
"diff". It is needed because attribute macros can't be negated,
and some users may want to force git into treating certain files as
text which are not recognized by the internal logic.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
---
This gives the user the chance to mess up completely (given an
"appropriate" setting of autocrlf), but I still think it's a good idea
to have that "-binary" built-in. attributes aren't used by noobs anyways.
In many cases, "diff" might be preferred over "text". Should I add a warning
to the doc?
Michael
Documentation/gitattributes.txt | 4 +++-
attr.c | 1 +
2 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 2694559..2a00f8c 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -510,7 +510,9 @@ the same time. The system knows a built-in attribute macro, `binary`:
which is equivalent to the above. Note that the attribute macros can only
be "Set" (see the above example that sets "binary" macro as if it were an
ordinary attribute --- setting it in turn unsets "crlf" and "diff").
-
+Therefore, there is also a built-in attribute macro `text` which allows
+you to mark certain files as text which git does not recognize automatically.
+It is equivalent to setting `crlf diff`.
DEFINING ATTRIBUTE MACROS
-------------------------
diff --git a/attr.c b/attr.c
index 17f6a4d..63e2837 100644
--- a/attr.c
+++ b/attr.c
@@ -283,6 +283,7 @@ static void free_attr_elem(struct attr_stack *e)
static const char *builtin_attr[] = {
"[attr]binary -diff -crlf",
+ "[attr]text diff crlf",
NULL,
};
--
1.6.0.3.514.g2f91b
^ permalink raw reply related
* Re: [PATCH 3/5] refactor userdiff textconv code
From: Jeff King @ 2008-10-24 13:51 UTC (permalink / raw)
To: Johannes Sixt; +Cc: git, Junio C Hamano, Matthieu Moy
In-Reply-To: <4901762A.3090003@viscovery.net>
On Fri, Oct 24, 2008 at 09:15:54AM +0200, Johannes Sixt wrote:
> +cat >expect.typechange <<'EOF'
> +Binary files a/file and /dev/null differ
> +diff --git a/file b/file
> +new file mode 120000
> +index ad8b3d2..67be421
> +--- /dev/null
> ++++ b/file
> +@@ -0,0 +1 @@
> ++frotz
> +\ No newline at end of file
> +EOF
Actually, I don't think this is right. The typechange has been broken
into two parts: the removal of the file contents and the addition of the
symlink. So the first part _should_ use the textconv, since it is just
comparing the file contents to /dev/null. But the second part should
not, since it is by definition just the text of the symlink. Ditto for
gitlinks, which have a special text representation. So how about this?
-- >8 --
only textconv regular files
We treat symlinks as text containing the results of the
symlink, so it doesn't make much sense to text-convert them.
Similarly gitlink components just end up as the text
"Subproject commit $sha1", which we should leave intact.
Note that a typechange may be broken into two parts: the
removal of the old part and the addition of the new. In that
case, we _do_ show the textconv for any part which is the
addition or removal of a file we would ordinarily textconv,
since it is purely acting on the file contents.
Signed-off-by: Jeff King <peff@peff.net>
---
diff.c | 3 +++
t/t4030-diff-textconv.sh | 28 ++++++++++++++++++++++++++++
2 files changed, 31 insertions(+), 0 deletions(-)
diff --git a/diff.c b/diff.c
index 52feba7..ce1317d 100644
--- a/diff.c
+++ b/diff.c
@@ -289,6 +289,9 @@ static int diff_filespec_textconv(struct diff_filespec *one)
if (!DIFF_FILE_VALID(one))
return 0;
+ if (!S_ISREG(one->mode))
+ return 0;
+
diff_filespec_load_driver(one);
if (!one->driver->textconv)
return 0;
diff --git a/t/t4030-diff-textconv.sh b/t/t4030-diff-textconv.sh
index e456746..a5cd99b 100755
--- a/t/t4030-diff-textconv.sh
+++ b/t/t4030-diff-textconv.sh
@@ -19,6 +19,22 @@ cat >expect.text <<'EOF'
+1
EOF
+cat >expect.typechange <<'EOF'
+--- a/file
++++ /dev/null
+@@ -1,2 +0,0 @@
+-0
+-1
+diff --git a/file b/file
+new file mode 120000
+index ad8b3d2..67be421
+--- /dev/null
++++ b/file
+@@ -0,0 +1 @@
++frotz
+\ No newline at end of file
+EOF
+
cat >hexdump <<'EOF'
#!/bin/sh
perl -e '$/ = undef; $_ = <>; s/./ord($&)/ge; print $_' "$1"
@@ -75,4 +91,16 @@ test_expect_success 'format-patch produces binary' '
test_cmp expect.binary actual
'
+# make a symlink the hard way that works on symlink-challenged file systems
+test_expect_failure 'textconv does not act on symlinks' '
+ echo -n frotz > file &&
+ git add file &&
+ git ls-files -s | sed -e s/100644/120000/ |
+ git update-index --index-info &&
+ git commit -m typechange &&
+ git show >diff &&
+ find_diff <diff >actual &&
+ test_cmp expect.typechange actual
+'
+
test_done
--
1.6.0.3.523.ge05eb.dirty
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox