Re: [PATCH v2] convert: add "status=delayed" to filter process protocol

[Lars Schneider <larsxschneider@gmail.com>]

> On 27 Feb 2017, at 11:53, Jeff King <peff@peff.net> wrote:
> 
> On Mon, Feb 27, 2017 at 11:32:47AM +0100, Lars Schneider wrote:
> 
>> ...
> 
>>> From Git's side, the loop is something like:
>>> 
>>> while (delayed_items > 0) {
>>> 	/* issue a wait, and get back the status/index pair */
>>> 	status = send_wait(&index);
>>> 	delayed_items--;
>>> 
>>> 	/*
>>> 	 * use "index" to find the right item in our list of files;
>>> 	 * the format can be opaque to the filter, so we could index
>>> 	 * it however we like. But probably numeric indices in an array
>>> 	 * are the simplest.
>>> 	 */
>>> 	assert(index > 0 && index < nr_items);
>>> 	item[index].status = status;
>>> 	if (status == SUCCESS)
>>> 		read_content(&item[index]);
>>> }
>>> 
>>> and the filter side just attaches the "index" string to whatever its
>>> internal queue structure is, and feeds it back verbatim when processing
>>> that item finishes.
>> 
>> That could work! I had something like that in mind:
>> 
>> I teach Git a new command "list_completed" or similar. The filter
>> blocks this call until at least one item is ready for Git. 
>> Then the filter responds with a list of paths that identify the
>> "ready items". Then Git asks for these ready items just with the
>> path and not with any content. Could that work? Wouldn't the path
>> be "unique" to identify a blob per filter run?
> 
> I think that could work, though I think there are few minor downsides
> compared to what I wrote above:
> 
>  - if you respond with "these items are ready", and then make Git ask
>    for each again, it's an extra round-trip for each set of ready
>    items. You could just say "an item is ready; here it is" in a single
>    response. For a local pipe the latency is probably negligible,
>    though.

It is true that the extra round-trip is not strictly necessary but I think
it simplifies the protocol/the code as I can reuse the convert machinery 
as is.


>  - using paths as the index would probably work, but it means Git has
>    to use the path to find the "struct checkout_entry" again. Which
>    might mean a hashmap (though if you have them all in a sorted list,
>    I guess you could also do a binary search).

Agreed. I changed my implementation to use an index following your
suggestion.

>  - Using an explicit index communicates to the filter not only what the
>    index is, but also that Git is prepared to accept a delayed response
>    for the item. For backwards compatibility, the filter would probably
>    advertise "I have the 'delayed' capability", and then Git could
>    choose to use it or not on a per-item basis. Realistically it would
>    not change from item to item, but rather operation to operation. So
>    that means we can easily convert the call-sites in Git to the async
>    approach incrementally. As each one is converted, it turns on the
>    flag that causes the filter code to send the "index" tag.

Agreed. I change the implementation accordingly and I will send out the
patches shortly.

Thanks,
Lars