[PATCH v4 0/9] Implement `promisor.storeFields` and `--filter=auto`

[Christian Couder <christian.couder@gmail.com>]

Introduction
============

A previous patch series added the possibility to pass additional
fields, a "partialCloneFilter" and a "token" for each advertised
promisor remote, from a server to a client through the
"promisor-remote" capability.

On the client side though, it has so far only been possible to use
this new information to compare it with local information and then
decide if the corresponding advertised promisor remote is accepted or
not.

For the "token" it would be useful if it could be stored on the
client. For example in a setup where the client uses specialized
remote helpers which need a token to access the promisor remotes
advertised by the server, storing the token would allow the token to
be used when the client directly accesses a promisor remote for
example to lazy fetch some blobs it now needs.

To enable such a workflow, where the server can rotate tokens and the
client can have updated tokens from the server by simply fetching from
it, the first part of this series introduces a new
"promisor.storeFields" configuration option on the client side,
similar to the "promisor.checkFields" configuration option. When field
names, "token" or "partialCloneFilter", are listed in this new
configuration option, then the values of these field names transmitted
by the server are stored in the local configuration on the client
side.

Note that for security reasons, the corresponding remote name and url
of the advertised promisor remotes must have already been configured
on the client side. No new remote name nor url are configured.

For the "partialCloneFilter" field, simply storing the value is not
enough to enable dynamic updates. Currently, when a user initiates a
partial clone with `--filter=<filter-spec>`, that specific
<filter-spec> is saved in the client's local configuration (e.g.,
remote.origin.partialCloneFilter). Subsequent fetches then reuse this
value, ignoring suggestions from the server.

To avoid breaking this mechanism and still be able to use the
<filter-spec> that the server suggests for the promisor remotes that
the client accepts, the second part of this series introduces a new
`--filter=auto` mode for `git clone` and `git fetch`.

When `--filter=auto` is used, then "auto" is still saved as the
<filter-spec> for the server locally on the client, and then when a
fetch-pack happens, instead of passing just "auto", the actual filter
requested by the client is computed by combining the <filter-spec>s
that the server suggested for the promisor remotes that the client
accepted. This uses the "combine" filter mechanism that already exists
in "list-objects-filter-options.{c,h}".

This way by just using `--filter=auto` when cloning, a client makes
sure it will use the <filter-spec>s suggested by the server for the
promisor remotes it accepts.

This work is part of the "LOP" effort documented in:

  Documentation/technical/large-object-promisors.adoc

See that doc for more information on the broader context.

Overview of the patches
=======================

Patches 1/9 and 2/9 are the first part of the series and implement the
new "promisor.storeFields" configuration option. Patch 1/9 is a small
preparatory refactoring.

Patches from 3/9 to 9/9 implement the `--filter=auto` option:

  - Patches 3/9 and 4/9 are cleanups of "builtin/clone.c" and
    "builtin/fetch.c" respectively that make the `filter_options`
    variable local to cmd_clone() or cmd_fetch().

  - Patch 5/9 is a doc update as `--filter=<filter-spec>` wasn't
    documented for `git fetch`.

  - Patch 6/9 improves "list-objects-filter-options.{c,h}" to
    support the new 'auto' mode.

  - Patches 7/9 and 8/9 improves "promisor-remote.{c,h}" to support
    the new 'auto' mode.

  - Patch 9/9 make the new 'auto' mode actually work by wiring up
    everything together.

CI Report
=========

All the tests pass, see:

https://github.com/chriscool/git/actions/runs/22059799525

Changes since v3
================

Thanks to Patrick Steinhardt, Jean-Noël Avila, Peff and Junio Hamano
for reviewing or commenting on the previous version!

The only change compared to v3 is in patch 6/9 where in
"list-objects-filter-options.c" the `allow_auto_filter` variable in
`list_objects_filter_release()` is now initialized after the check to
return if `filter_options` is NULL instead of before that check.

Range diff since v3
===================

 1:  79255ceba7 =  1:  79255ceba7 promisor-remote: refactor initialising field lists
 2:  012aa7ef19 =  2:  012aa7ef19 promisor-remote: allow a client to store fields
 3:  f17a62e73e =  3:  f17a62e73e clone: make filter_options local to cmd_clone()
 4:  3c6e28dd84 =  4:  3c6e28dd84 fetch: make filter_options local to cmd_fetch()
 5:  3037d546b2 =  5:  3037d546b2 doc: fetch: document `--filter=<filter-spec>` option
 6:  9ce57b88dc !  6:  366c93e836 list-objects-filter-options: support 'auto' mode for --filter
    @@ list-objects-filter-options.c: void list_objects_filter_release(
        struct list_objects_filter_options *filter_options)
      {
        size_t sub;
    -+  unsigned int allow_auto_filter = filter_options->allow_auto_filter;
    ++  unsigned int allow_auto_filter;
      
        if (!filter_options)
                return;
    -@@ list-objects-filter-options.c: void list_objects_filter_release(
    ++
    ++  allow_auto_filter = filter_options->allow_auto_filter;
    +   strbuf_release(&filter_options->filter_spec);
    +   free(filter_options->sparse_oid_name);
    +   for (sub = 0; sub < filter_options->sub_nr; sub++)
                list_objects_filter_release(&filter_options->sub[sub]);
        free(filter_options->sub);
        list_objects_filter_init(filter_options);
 7:  37042f7019 =  7:  2eb3b9cddd promisor-remote: keep advertised filters in memory
 8:  dd17069aad =  8:  fae3e9089d promisor-remote: change promisor_remote_reply()'s signature
 9:  0f9675f477 =  9:  4627d513d6 fetch-pack: wire up and enable auto filter logic


Christian Couder (9):
  promisor-remote: refactor initialising field lists
  promisor-remote: allow a client to store fields
  clone: make filter_options local to cmd_clone()
  fetch: make filter_options local to cmd_fetch()
  doc: fetch: document `--filter=<filter-spec>` option
  list-objects-filter-options: support 'auto' mode for --filter
  promisor-remote: keep advertised filters in memory
  promisor-remote: change promisor_remote_reply()'s signature
  fetch-pack: wire up and enable auto filter logic

 Documentation/config/promisor.adoc           |  33 +++
 Documentation/fetch-options.adoc             |  19 ++
 Documentation/git-clone.adoc                 |  25 +-
 Documentation/gitprotocol-v2.adoc            |  24 +-
 Makefile                                     |   1 +
 builtin/clone.c                              |  18 +-
 builtin/fetch.c                              |  50 ++--
 connect.c                                    |   3 +-
 fetch-pack.c                                 |  24 ++
 list-objects-filter-options.c                |  39 ++-
 list-objects-filter-options.h                |   6 +
 list-objects-filter.c                        |   8 +
 promisor-remote.c                            | 256 +++++++++++++++++--
 promisor-remote.h                            |  17 +-
 t/meson.build                                |   1 +
 t/t5710-promisor-remote-capability.sh        | 123 +++++++++
 t/unit-tests/u-list-objects-filter-options.c |  53 ++++
 transport.c                                  |   1 +
 18 files changed, 628 insertions(+), 73 deletions(-)
 create mode 100644 t/unit-tests/u-list-objects-filter-options.c

-- 
2.53.0.77.g4627d513d6