[PATCH RFC 0/3] cat-file: make remote-object-info request in batches

[Pablo Sabater <pabloosabaterr@gmail.com>]

Currently the command remote-object-info of git cat-file has a hard
limit of 10k OIDs to query in the same command. So, if you want to query
100k objects for example you would have to run the command 10 times.

This seemed strange to me because git backfill for example has a
default limit of 50k objects and backfill fetches full objects, while
remote-object-info only requests metadata (size, type, both or none) so
it should be able to handle at least the same, if not more.

Even if we kept the 10k OID limit for remote-object-info, it is annoying
having to run the command multiple times, so I propose to allow as many
OIDs as wanted and batch them internally.

Regarding which number of objects should be the limit, I benchmarked
different batch sizes over a simulated 47ms RTT (details in patch 2):

+------------+---------------------+
|            | objects requested   |
| batch size |   100k   |   400k   |
+------------+----------+----------+
| 10k        |  1.691s  |  5.862s  |
| 50k        |  0.981s  |  2.397s  |
| 100k       |  0.915s  |  2.058s  |
| 200k       |  0.924s  |  1.889s  |
| 400k       |  0.908s  |  1.820s  |
+------------+----------+----------+

A noticeable difference can be seen from a 10k OID batch to a 50k batch.
After that the difference isn't as noticeable (the number of batches to
be requested isn't that big).

I propose 50k OIDs as the default number of OIDs for the
remote-object-info batches. Coincidentally, git backfill's default batch
size is 50k as well though it's unrelated.

I have barely done any benchmarking before, so I would appreciate
any feedback on the one in patch 2. Is there a better way to do it,
or other things I should take into account?

Should the server also check for a limit on the number of OIDs it
accepts in a single object-info request?

GitHub CI passed:
https://github.com/pabloosabaterr/git/actions/runs/40424912154

Signed-off-by: Pablo Sabater <pabloosabaterr@gmail.com>
---
Pablo Sabater (3):
      cat-file: extract the transport ownership from get_remote_info()
      cat-file: make remote-object-info request in batches
      cat-file: add --remote-max-batch option

 Documentation/git-cat-file.adoc        |  16 ++-
 builtin/cat-file.c                     | 174 +++++++++++++++++++--------------
 t/t1017-cat-file-remote-object-info.sh |  68 +++++++++++++
 transport-helper.c                     |  13 ++-
 transport-internal.h                   |   6 +-
 transport.c                            |  33 +++----
 transport.h                            |   6 +-
 7 files changed, 208 insertions(+), 108 deletions(-)


---
base-commit: 37b3475fb530c14109211faca33716a173db61ed
change-id: 20261015-remote-object-info-batch-c65f13fb5a95