[PATCH 00/11] [RFC] config-batch: a new builtin for tools querying config

["Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>]

This RFC explores a new git config-batch builtin that allows tools to
interact with Git's config data with multiple queries using a single
process. This is an orthogonal alternative to the effort to create a stable,
linkable config API. Both approaches have different strengths.

My main motivation is the performance of git-credential-manager on Windows
platforms as it can call git config get dozens of times. At 150-200ms per
execution, that adds up significantly, leading to multiple seconds just to
load a credential that already exists. I believe that there are other
benefits to having this interface available, but I can't recall any
specifics at the moment.

This RFC adds git config-batch with a protocol over stdin/stdout for
executing multiple config queries. The implementation has a limited set of
potential queries, but also creates a model for compatibility for tools to
automatically adapt to different Git versions.

I'm submitting this as an RFC before I've polished all of the details
because I want to make sure I'm going down a good direction. Please focus
feedback in these questions:

 * Is this a worthwhile feature to add to Git?
 * Is this a reasonable protocol for stdin/stdout?
 * How can we structure the code to make it easier to contribute new
   commands in the future?
 * This seems like a place where parallel contributions can be made once the
   baseline is implemented. Is there interest in further contributions to
   expand the commands?

This RFC adds the following commands over stdin:

 1. help lists the available commands, giving the caller an understanding of
    what is available in this Git version.
 2. get loads a value for a given key within a certain scope, with optional
    value patterns.
 3. set assigns a key-value pair in a given scope.
 4. unset removes a key-value pair in a given scope with optional value
    patterns.

Each command has an associated version, in case we need to expand or alter
the functionality in the future. This includes the potential to deprecate
and remove certain versions that we no longer want to support, such as
replacing set version 1 with a version 2 and making version 1 no longer
available. I do hope that we will mostly be able to move with new command
names, such as a set-all command including the options for git config set
--all ... instead of increasing the version of the set command.

There is a -z option that changes the command interface to use
NUL-terminated strings. Two NULs specify a command boundary, which promotes
compatibility with a caller that sends an unknown command. However, this
means that we cannot specify an empty string as a token within a command
unless we add more data. This format uses <N>:<string> to provide the
integer <N> which specifies the length of <string>. This is a little
cumbersome, but the format is intended for tools, not humans.

I have a test integration with git-credential-manager available [1] for
testing. This includes a model for interacting with git config-batch in a
compatible way that will respond to certain features not being available:

 1. If git config-batch fails immediately, then all queries are handled by
    git config.
 2. If git config-batch starts without failure, then the first query is for
    the help command.
 3. As queries come to the config system, the query is checked against the
    available commands advertised by git config-batch. If the appropriate
    command is available, then the query is made in that process. If not,
    then the query uses the existing git config command.

One thing that I think would be valuable to include is a reload command that
signals that the git config-batch process should reload the configset into
memory due to config manipulations in other processes, especially while git
config-batch doesn't have all capabilities from git config. I'll include
that in the first version for review, if this RFC leads to positive support.

[1] https://github.com/git-ecosystem/git-credential-manager/pull/2245

I have a few concerns with this implementation that I'd like to improve
before submitting a version for full review. I list them here so you can see
the flaws that I already see, but also so you can add to this list:

 * We need a reload command (as mentioned above).
 * The tests need to include a submodule and submodule-level config.
 * When specifying the local scope to the get command, the matched value
   does not include worktree or submodule config in the same way that git
   config get --local <key> would.
 * The token-parsing API in this helper is still too complicated to use. I
   should create parsing tooling similar to the parse-opts API so each
   command could specify its use of positional values and optional
   arguments.
 * The use of arg:<arg> to specify an optional argument creates the
   inability to submit a value that starts with arg:. Consider alternative
   ways to specify arguments or to specify that the remaining data in the
   command (including spaces) is a final positional argument.
 * In general, I found myself implementing behavior based on the deprecated
   forms of git config that use the --get or --unset style arguments instead
   of git config (set|unset|get) subcommands. It's worth making sure that
   any references to equivalent git config commands use the new modes.
 * I need to add an --[no-]includes option as a command-line argument that
   signals whether include sections should be followed. I don't believe this
   should be specified on a per-command basis, but I'm open to suggestions.
 * I have an early draft of a technical document detailing the plan for this
   builtin. It has some lists of intended future commands that have not been
   implemented. This would also be a good place to document any parsing APIs
   built to help contributors adding to this builtin.

Thanks, -Stolee

Derrick Stolee (11):
  config-batch: basic boilerplate of new builtin
  config-batch: create parse loop and unknown command
  config-batch: implement get v1
  config-batch: create 'help' command
  config-batch: add NUL-terminated I/O format
  docs: add design doc for config-batch
  config: extract location structs from builtin
  config-batch: pass prefix through commands
  config-batch: add 'set' v1 command
  t1312: create read/write test
  config-batch: add unset v1 command

 .gitignore                                |   1 +
 Documentation/git-config-batch.adoc       | 214 ++++++
 Documentation/meson.build                 |   1 +
 Documentation/technical/config-batch.adoc |  70 ++
 Makefile                                  |   1 +
 builtin.h                                 |   7 +
 builtin/config-batch.c                    | 772 ++++++++++++++++++++++
 builtin/config.c                          | 117 +---
 command-list.txt                          |   1 +
 config.c                                  | 116 ++++
 config.h                                  |  26 +
 git.c                                     |   1 +
 meson.build                               |   1 +
 t/meson.build                             |   1 +
 t/t1312-config-batch.sh                   | 372 +++++++++++
 15 files changed, 1592 insertions(+), 109 deletions(-)
 create mode 100644 Documentation/git-config-batch.adoc
 create mode 100644 Documentation/technical/config-batch.adoc
 create mode 100644 builtin/config-batch.c
 create mode 100755 t/t1312-config-batch.sh


base-commit: 83a69f19359e6d9bc980563caca38b2b5729808c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2033%2Fderrickstolee%2Fbatched-config-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2033/derrickstolee/batched-config-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2033
-- 
gitgitgadget