* Re: Something is broken in repack. Why not with fork and pipes?
@ 2007-12-12 18:47 J.C. Pizarro
2007-12-12 19:41 ` Johannes Schindelin
0 siblings, 1 reply; 2+ messages in thread
From: J.C. Pizarro @ 2007-12-12 18:47 UTC (permalink / raw)
To: Linus Torvalds, Andreas Ericsson
Cc: David Miller, Nicolas Pitre, jonsmirl, Junio C Hamano, gcc, git
At http://gcc.gnu.org/ml/gcc/2007-12/msg00360.html, Andreas Ericsson
<ae@op5.se> wrote:
> If it's still an issue next week, we'll have a 16 core (8 dual-core cpu's)
> machine with some 32gb of ram in that'll be free for about two days.
> You'll have to remind me about it though, as I've got a lot on my mind
> these days.
>
>
> --
> Andreas Ericsson andreas.ericsson@op5.se
> OP5 AB www.op5.se
> Tel: +46 8-230225 Fax: +46 8-230231
It's good idea if it's for 24/365.25 that it does
autorepack-compute-again-again-again-those-unexplored-deltas of
git repositories in realtime. :D
Some body can do "git clone" that it could give smaller that one hour ago :D
-----------------------------------------------------------------
To Linus, Why don't you forget the threaded implementation of your repo-pack?
To imagine a "buggy bloated threading implementation originated to try it to
work only in HyperThreading Intel CPUs and 8 cores x 8 threads/core
Niagara Sparcs"
IMHO, in multicored machine, multiprocessed implementation of repo-pack perfomes
better than multithreaded implementation, although i've not their results.
It has not issue, not problem, etc. with memory allocation of threads,
so monothreaded memory allocation is simple and fast!
You can see "Why not with fork and pipes like in linux?" at
http://gcc.gnu.org/ml/gcc/2007-12/msg00203.html
http://gcc.gnu.org/ml/gcc/2007-12/msg00209.html
For easy implementation, don't use threads due to complicated condition races
between threads of multithreaded processes.
To use only condition races between monothreaded processes with select/epoll
only in the parent process. It's due to the KISS principle works.
The children processes share almost readed-only memory due to COW
(Copy On Write), so, before forking, the parent must to have a large
plain data structures in C for children. The children use pipes to
realize a complex intercommunication that the parent updates the
results computated by the children almost of the time.
Another implementation is that the children can realize a locked
load-and-store to/from unique filesystem's database if big memory to
store data is a big problem.
Another implementation is to consider children processes as intensive-CPU
slaves and parent process as the master that manipulates the big database.
If you want to measure the performance between multiprocessed vs multithreaded
implementation of repo-pack then you have to remember that
For same data input size and same data output size, to get the
seconds of your wall-clock or watch-clock as a measure of the benchmark
of this repo-pack.
The numeric data posted to mailing list about the timings dependently of # of
threads are bad measured because they don't say how is small the result repo.
and don't say if the results are the same independently of # of threads.
For good measures, we need "to plot the curves", e.g. based in
( # of threads, elapsed time of wall-clock, data input size, data output size )
and we can observe the intersection between above curves.
J.C.Pizarro
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Something is broken in repack. Why not with fork and pipes?
2007-12-12 18:47 Something is broken in repack. Why not with fork and pipes? J.C. Pizarro
@ 2007-12-12 19:41 ` Johannes Schindelin
0 siblings, 0 replies; 2+ messages in thread
From: Johannes Schindelin @ 2007-12-12 19:41 UTC (permalink / raw)
To: J.C. Pizarro
Cc: Linus Torvalds, Andreas Ericsson, David Miller, Nicolas Pitre,
jonsmirl, Junio C Hamano, gcc, git
Hi,
On Wed, 12 Dec 2007, J.C. Pizarro wrote:
> It's good idea if it's for 24/365.25 that it does
> autorepack-compute-again-again-again-those-unexplored-deltas of
> git repositories in realtime. :D
This sentence does not parse.
> Some body can do "git clone" that it could give smaller that one hour ago :D
Neither does this.
> To Linus, Why don't you forget the threaded implementation of your
> repo-pack?
Please do a little research before you ask such questions: it is neither
Linus who did it, nor is it better to use processes than threads.
Besides, your proposal has nothing to do with the issue of this thread
(memory consumption).
Ciao,
Dscho
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2007-12-12 19:42 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-12-12 18:47 Something is broken in repack. Why not with fork and pipes? J.C. Pizarro
2007-12-12 19:41 ` Johannes Schindelin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).