Re: [PATCH] Add fetch.recurseSubmoduleParallelism config option

[Junio C Hamano <gitster@pobox.com>]

Stefan Beller <sbeller@google.com> writes:

>> The parallel_process API could learn a new "verbose" feature that it
>> by itself shows some messages like
>>
>>     "processing the 'frotz' job with N tasks"
>>     "M tasks finished (N still running)"
>
> I know what to fill in for M and N, 'frotz' is a bit unclear to me.

At least I don't know what M and N should be, and I'm curious how
you'll define them.  See below.

>> in the output stream from strategic places.  For example, the first
>> message will come at the end of pp_init(), and the second message
>> will be appended at the end of buffered output of a task that has
>> just been finished.  Once you have something like that, you could
>> check for them in a test in t/.
>>
>> Just a thought.
>
> I like that thought. :)


A few more random thoughts:

 * The only thing you could rely on if you were to use the above in
   your tests is the one from pp_init() that declares how many
   processes the machinery is going to use.  M/N will be unstable,
   depending on the scheduling order (e.g. the foreground process
   may take a lot of time to finish, while many other processes
   finish first).

 * Every time the foreground process (i.e. the one whose output is
   tee-ed to the overall output from the machinery) finishes, you
   can emit "M tasks finished (N still running)", but I am not sure
   what M should be.  It is debatable how to account for background
   processes that have already completed but whose output haven't
   been shown.

   One school of thought that is in line with the "pretend as if the
   background tasks are started immediately after the foreground
   task finishes, and they run at infinite speed and produce output
   in no time" idea, on which the "queue output from the background
   processes and emit at once in order to avoid intermixing" design
   was based on, would be not to include them in M (i.e. finished
   ones), because their output haven't been emitted and we are
   pretending that they haven't even been started.  If you take this
   approach, you however may have to include them in N (i.e. still
   running), but that would likely bump N beyond the maximum number
   of simultaneous processes.

   The other school of thought would of course tell the truth and
   include the number of finished background processes in M, as they
   have finished already in the reality.  This will not risk showing
   N that is beyond the maximum, but your first "progress" output
   might say "3 tasks finished", which will make it look odd in a
   different way.