* [PATCH bpf-next v3 0/2] bpftool: Batch bounded hash map dumps
@ 2026-09-11 2:51 Tianyi Chen
2026-09-11 2:51 ` [PATCH bpf-next v3 1/2] bpftool: Use batch lookups for " Tianyi Chen
2026-09-11 2:51 ` [PATCH bpf-next v3 2/2] selftests/bpf: Check bpftool batch map dump contents Tianyi Chen
0 siblings, 2 replies; 4+ messages in thread
From: Tianyi Chen @ 2026-09-11 2:51 UTC (permalink / raw)
To: qmo, bpf; +Cc: andrii, eddyz87, ihor.solodrai, linux-kselftest
Use lookup batches for ordinary hash maps whose maximum key/value
storage fits within 4 MiB. The second patch checks complete output around
the batch boundary, short keys, odd-sized values and BTF formatting.
Changes in v3:
- Explain the batching benefit, conservative memory bound, and why
ineligible maps retain the existing traversal in the commit message.
- Rebase onto current bpf-next and use Assisted-by: LLM.
The v2 implementation and tests are unchanged. Fatal batch diagnostics
remain on stderr so JSON element arrays contain only map entries.
v2: https://lore.kernel.org/r/178874362960.1415955.9584865244247659247.batch-v2-0@tychen.cc
Request: https://github.com/libbpf/bpftool/issues/63
The previous version's CI PR expired after repeated "Patch is empty"
reports. The revised series applies cleanly to the current tree and is
sent as a new thread using git format-patch and git send-email.
Validation:
- Built bpftool and its manual pages, and built the focused selftest
runner with LLVM 20.
- All 11 bpftool_map_batch subtests passed with no skips or failures
in an x86-64 KVM guest running Linux 7.3.0-rc2 from current bpf/master.
- The four bpftool series were tested together: all 69 subtests passed.
The cpumask suite (36 subtests) and test_maps also passed with no skips.
- Bash syntax and the existing bpftool synchronization checks passed.
This was a focused run; unrelated selftests requiring unavailable kernel
features were excluded from the build with PERMISSIVE=1.
Tianyi Chen (2):
bpftool: Use batch lookups for bounded hash map dumps
selftests/bpf: Check bpftool batch map dump contents
tools/bpf/bpftool/map.c | 116 ++++++++++-
.../bpf/prog_tests/bpftool_map_batch.c | 187 ++++++++++++++++++
2 files changed, 295 insertions(+), 8 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpftool_map_batch.c
base-commit: af0b84a9215d951d16f26b7ee34353b970cf5d4e
--
2.55.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH bpf-next v3 1/2] bpftool: Use batch lookups for bounded hash map dumps
2026-09-11 2:51 [PATCH bpf-next v3 0/2] bpftool: Batch bounded hash map dumps Tianyi Chen
@ 2026-09-11 2:51 ` Tianyi Chen
2026-09-11 3:37 ` bot+bpf-ci
2026-09-11 2:51 ` [PATCH bpf-next v3 2/2] selftests/bpf: Check bpftool batch map dump contents Tianyi Chen
1 sibling, 1 reply; 4+ messages in thread
From: Tianyi Chen @ 2026-09-11 2:51 UTC (permalink / raw)
To: qmo, bpf; +Cc: andrii, eddyz87, ihor.solodrai, linux-kselftest
From: Tianyi Chen <hi@tychen.cc>
Use BPF_MAP_LOOKUP_BATCH when dumping hash maps to reduce the number
of BPF syscalls while preserving plain, JSON and BTF formatting.
For a 100,000-entry hash map in an x86-64 KVM guest, BPF syscall counts
fell from 200,004 to 395. The median of five untraced runs fell from
0.774013 s to 0.733844 s, about 5.2% in this measurement. Syscall counts
were collected separately with strace.
Hash batch lookup must fit an entire bucket. Restrict eligibility to
maps whose maximum key/value storage fits in 4 MiB, so even a worst-case
bucket can fit without restarting a partially printed dump. Start with
up to 256 entries and grow on ENOSPC using the same input cursor.
Fall back to individual lookups only when the initial batch operation
is unsupported. Restarting after output has begun would duplicate
entries. Process the final partial batch on ENOENT, but do not trust
count or output buffers after other errors. Keep fatal diagnostics on
stderr so JSON element arrays contain only map entries.
Link: https://github.com/libbpf/bpftool/issues/63
Assisted-by: LLM
Signed-off-by: Tianyi Chen <hi@tychen.cc>
---
tools/bpf/bpftool/map.c | 116 +++++++++++++++++++++++++++++++++++++---
1 file changed, 108 insertions(+), 8 deletions(-)
diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c
index 684a8fb72414..ea6f1389245f 100644
--- a/tools/bpf/bpftool/map.c
+++ b/tools/bpf/bpftool/map.c
@@ -740,15 +740,10 @@ static int do_show(int argc, char **argv)
return errno == ENOENT ? 0 : -1;
}
-static int dump_map_elem(int fd, void *key, void *value,
- struct bpf_map_info *map_info, struct btf *btf,
- json_writer_t *btf_wtr)
+static void print_map_elem(void *key, void *value,
+ struct bpf_map_info *map_info, struct btf *btf,
+ json_writer_t *btf_wtr)
{
- if (bpf_map_lookup_elem(fd, key, value)) {
- print_entry_error(map_info, key, errno);
- return -1;
- }
-
if (json_output) {
print_entry_json(map_info, key, value, btf);
} else if (btf) {
@@ -762,10 +757,111 @@ static int dump_map_elem(int fd, void *key, void *value,
} else {
print_entry_plain(map_info, key, value);
}
+}
+
+static int dump_map_elem(int fd, void *key, void *value,
+ struct bpf_map_info *map_info, struct btf *btf,
+ json_writer_t *btf_wtr)
+{
+ if (bpf_map_lookup_elem(fd, key, value)) {
+ print_entry_error(map_info, key, errno);
+ return -1;
+ }
+ print_map_elem(key, value, map_info, btf, btf_wtr);
return 0;
}
+#define MAP_DUMP_BATCH_SIZE 256U
+#define MAP_DUMP_BATCH_MAX_BYTES (4 * 1024 * 1024)
+
+/* Return 1 to use individual lookups, but only before batch traversal starts. */
+static int dump_map_batch(int fd, void *key, void *value,
+ struct bpf_map_info *info, struct btf *btf,
+ json_writer_t *wtr, unsigned int *num_elems)
+{
+ __u32 capacity, count, batch = 0, next_batch = 0, i;
+ void *keys = NULL, *values = NULL, *buf;
+ bool first = true, can_fallback = true;
+ int err;
+
+ /*
+ * Hash lookup batches must accommodate a whole bucket. Restrict the
+ * optimization to maps whose worst-case bucket fits the memory budget,
+ * so a later ENOSPC never forces a restart after printing some entries.
+ * Division also bounds the allocation multiplications on 32-bit hosts.
+ */
+ if (info->type != BPF_MAP_TYPE_HASH || !info->max_entries ||
+ (__u64)info->key_size + info->value_size >
+ MAP_DUMP_BATCH_MAX_BYTES / info->max_entries)
+ return 1;
+
+ capacity = min(info->max_entries, MAP_DUMP_BATCH_SIZE);
+resize:
+ buf = realloc(keys, (size_t)capacity * info->key_size);
+ if (!buf) {
+ err = ENOMEM;
+ goto error;
+ }
+ keys = buf;
+ buf = realloc(values, (size_t)capacity * info->value_size);
+ if (!buf) {
+ err = ENOMEM;
+ goto error;
+ }
+ values = buf;
+
+ while (true) {
+ count = capacity;
+ err = bpf_map_lookup_batch(fd, first ? NULL : &batch,
+ &next_batch, keys, values, &count, NULL);
+ err = err ? errno : 0;
+ /*
+ * Older kernels reject the command before updating count. Do not
+ * inspect the buffers on these errors, or fall back after progress.
+ */
+ if (can_fallback && (err == EINVAL || err == EOPNOTSUPP ||
+ err == 524 /* ENOTSUPP */)) {
+ err = 1;
+ goto out;
+ }
+ can_fallback = false;
+ if (err == ENOSPC) {
+ if (capacity == info->max_entries)
+ goto error;
+ capacity += min(capacity, info->max_entries - capacity);
+ /* Preserve the input cursor: the oversized bucket was not read. */
+ goto resize;
+ }
+ /* In particular, EFAULT can leave count and the buffers invalid. */
+ if (err && err != ENOENT)
+ goto error;
+ for (i = 0; i < count; i++) {
+ /*
+ * Keep the alignment provided by individual lookups, including
+ * for BTF types whose map key/value size is not aligned.
+ */
+ memcpy(key, keys + (size_t)i * info->key_size, info->key_size);
+ memcpy(value, values + (size_t)i * info->value_size, info->value_size);
+ print_map_elem(key, value, info, btf, wtr);
+ (*num_elems)++;
+ }
+ if (err == ENOENT) {
+ err = 0;
+ goto out;
+ }
+ first = false;
+ batch = next_batch;
+ }
+error:
+ fprintf(stderr, "Error: can't lookup map batch: %s\n", strerror(err));
+ err = -1;
+out:
+ free(keys);
+ free(values);
+ return err;
+}
+
static int maps_have_btf(int *fds, int nb_fds)
{
struct bpf_map_info info = {};
@@ -869,6 +965,9 @@ map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr,
p_info("Warning: cannot read values from %s map with value_size != 8",
map_type_str);
}
+ err = dump_map_batch(fd, key, value, info, btf, wtr, &num_elems);
+ if (err != 1)
+ goto end_dump;
while (true) {
err = bpf_map_get_next_key(fd, prev_key, key);
if (err) {
@@ -881,6 +980,7 @@ map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr,
prev_key = key;
}
+end_dump:
if (wtr) {
jsonw_end_array(wtr); /* elements */
if (show_header)
--
2.55.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH bpf-next v3 2/2] selftests/bpf: Check bpftool batch map dump contents
2026-09-11 2:51 [PATCH bpf-next v3 0/2] bpftool: Batch bounded hash map dumps Tianyi Chen
2026-09-11 2:51 ` [PATCH bpf-next v3 1/2] bpftool: Use batch lookups for " Tianyi Chen
@ 2026-09-11 2:51 ` Tianyi Chen
1 sibling, 0 replies; 4+ messages in thread
From: Tianyi Chen @ 2026-09-11 2:51 UTC (permalink / raw)
To: qmo, bpf; +Cc: andrii, eddyz87, ihor.solodrai, linux-kselftest
From: Tianyi Chen <hi@tychen.cc>
Exercise hash map dumps around the initial batch size and across
multiple batches, including empty and single-entry maps. Compare each
complete unordered key/value set with the input data in plain, JSON
and pretty JSON output.
Cover one-byte keys, three-byte values and BTF-formatted maps to catch
cursor sizing, buffer alignment and formatting regressions.
Assisted-by: LLM
Signed-off-by: Tianyi Chen <hi@tychen.cc>
---
.../bpf/prog_tests/bpftool_map_batch.c | 187 ++++++++++++++++++
1 file changed, 187 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpftool_map_batch.c
diff --git a/tools/testing/selftests/bpf/prog_tests/bpftool_map_batch.c b/tools/testing/selftests/bpf/prog_tests/bpftool_map_batch.c
new file mode 100644
index 000000000000..b4216ed778ef
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/bpftool_map_batch.c
@@ -0,0 +1,187 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <bpftool_helpers.h>
+#include <bpf/btf.h>
+#include <ctype.h>
+
+#define MAX_ENTRIES 1025
+#define RECORD_SIZE 256
+#define OUTPUT_SIZE (MAX_ENTRIES * RECORD_SIZE + 1024)
+
+struct dump_case {
+ const char *name;
+ unsigned int count;
+ unsigned int key_size;
+ unsigned int value_size;
+ bool btf;
+};
+
+static void hex_bytes(char *out, const void *data, unsigned int size, bool json)
+{
+ const unsigned char *bytes = data;
+ unsigned int i;
+
+ if (json)
+ *out++ = '[';
+ for (i = 0; i < size; i++) {
+ if (json && i)
+ *out++ = ',';
+ out += sprintf(out, json ? "\"0x%02x\"" : "%02x", bytes[i]);
+ }
+ if (json)
+ *out++ = ']';
+ *out = '\0';
+}
+
+static void expected_record(char *record, const struct dump_case *test,
+ unsigned int index, bool json)
+{
+ __u32 key = index, value = index * 37 + 11;
+ unsigned char short_key = index;
+ char key_hex[64], value_hex[64], formatted[96];
+
+ hex_bytes(key_hex, test->key_size == 1 ? (void *)&short_key : &key,
+ test->key_size, json);
+ hex_bytes(value_hex, &value, test->value_size, json);
+ snprintf(formatted, sizeof(formatted), "{\"key\":%u,\"value\":%u}",
+ key, value);
+ if (json && test->btf)
+ snprintf(record, RECORD_SIZE,
+ "{\"key\":%s,\"value\":%s,\"formatted\":%s}",
+ key_hex, value_hex, formatted);
+ else if (json)
+ snprintf(record, RECORD_SIZE, "{\"key\":%s,\"value\":%s}",
+ key_hex, value_hex);
+ else if (test->btf)
+ snprintf(record, RECORD_SIZE, "%s", formatted);
+ else
+ snprintf(record, RECORD_SIZE, "key:%svalue:%s", key_hex, value_hex);
+}
+
+static void check_dump(const struct dump_case *test, __u32 id, bool json, bool pretty)
+{
+ bool array = json || test->btf;
+ char command[MAX_BPFTOOL_CMD_LEN], expected[RECORD_SIZE], footer[64];
+ bool seen[MAX_ENTRIES] = {};
+ char *output, *src, *dst, *cursor;
+ unsigned int i, n;
+ int err;
+
+ output = calloc(1, OUTPUT_SIZE);
+ if (!ASSERT_OK_PTR(output, "alloc_output"))
+ return;
+ snprintf(command, sizeof(command), "%smap dump id %u",
+ pretty ? "-p " : json ? "-j " : "", id);
+ err = get_bpftool_command_output(command, output, OUTPUT_SIZE);
+ if (!ASSERT_OK(err, "map_dump"))
+ goto out;
+ /*
+ * Ignore presentation whitespace, but compare complete records and all
+ * punctuation. Expected contents come only from the input data, never
+ * from another map walk or bpftool invocation.
+ */
+ for (src = output, dst = output; *src; src++)
+ if (!isspace((unsigned char)*src))
+ *dst++ = *src;
+ *dst = '\0';
+ cursor = output;
+ if (array) {
+ if (!ASSERT_EQ(*cursor, '[', "array_start"))
+ goto out;
+ cursor++;
+ }
+ for (n = 0; n < test->count; n++) {
+ if (array && n) {
+ if (!ASSERT_EQ(*cursor, ',', "record_separator"))
+ goto out;
+ cursor++;
+ }
+ for (i = 0; i < test->count; i++) {
+ if (seen[i])
+ continue;
+ expected_record(expected, test, i, json);
+ if (!strncmp(cursor, expected, strlen(expected)))
+ break;
+ }
+ if (!ASSERT_LT(i, test->count, "unique_expected_record"))
+ goto out;
+ seen[i] = true;
+ cursor += strlen(expected);
+ }
+ if (array) {
+ ASSERT_STREQ(cursor, "]", "array_end_and_count");
+ } else {
+ snprintf(footer, sizeof(footer), "Found%uelement%s", test->count,
+ test->count == 1 ? "" : "s");
+ ASSERT_STREQ(cursor, footer, "plain_count");
+ }
+out:
+ free(output);
+}
+
+static void run_dump_case(const struct dump_case *test)
+{
+ LIBBPF_OPTS(bpf_map_create_opts, opts);
+ struct bpf_map_info info = {};
+ __u32 info_len = sizeof(info);
+ struct btf *btf = NULL;
+ unsigned int i;
+ int fd = -1;
+
+ if (test->btf) {
+ btf = btf__new_empty();
+ if (!ASSERT_OK_PTR(btf, "btf_new"))
+ return;
+ if (!ASSERT_EQ(btf__add_int(btf, "unsigned int", 4, 0), 1,
+ "btf_int") ||
+ !ASSERT_OK(btf__load_into_kernel(btf), "btf_load"))
+ goto out;
+ opts.btf_fd = btf__fd(btf);
+ opts.btf_key_type_id = 1;
+ opts.btf_value_type_id = 1;
+ }
+ fd = bpf_map_create(BPF_MAP_TYPE_HASH, "dump_batch", test->key_size,
+ test->value_size, test->count ?: 1, &opts);
+ if (!ASSERT_OK_FD(fd, "map_create"))
+ goto out;
+ for (i = 0; i < test->count; i++) {
+ __u32 key = i, value = i * 37 + 11;
+ unsigned char short_key = i;
+ void *key_ptr = test->key_size == 1 ? (void *)&short_key : &key;
+
+ if (!ASSERT_OK(bpf_map_update_elem(fd, key_ptr, &value, BPF_ANY),
+ "map_update"))
+ goto out;
+ }
+ if (!ASSERT_OK(bpf_map_get_info_by_fd(fd, &info, &info_len), "map_info"))
+ goto out;
+ check_dump(test, info.id, false, false);
+ check_dump(test, info.id, true, false);
+ check_dump(test, info.id, true, true);
+out:
+ if (fd >= 0)
+ close(fd);
+ btf__free(btf);
+}
+
+void test_bpftool_map_batch(void)
+{
+ static const struct dump_case cases[] = {
+ { "empty", 0, 4, 4 },
+ { "single", 1, 4, 4 },
+ { "below_batch", 255, 4, 4 },
+ { "exact_batch", 256, 4, 4 },
+ { "above_batch", 257, 4, 4 },
+ { "multiple_batches", 1025, 4, 4 },
+ { "one_byte_key", 256, 1, 4 },
+ { "odd_value_size", 257, 4, 3 },
+ { "btf_empty", 0, 4, 4, true },
+ { "btf_single", 1, 4, 4, true },
+ { "btf_multiple_batches", 1025, 4, 4, true },
+ };
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++)
+ if (test__start_subtest(cases[i].name))
+ run_dump_case(&cases[i]);
+}
--
2.55.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH bpf-next v3 1/2] bpftool: Use batch lookups for bounded hash map dumps
2026-09-11 2:51 ` [PATCH bpf-next v3 1/2] bpftool: Use batch lookups for " Tianyi Chen
@ 2026-09-11 3:37 ` bot+bpf-ci
0 siblings, 0 replies; 4+ messages in thread
From: bot+bpf-ci @ 2026-09-11 3:37 UTC (permalink / raw)
To: diannaaav, qmo, bpf
Cc: andrii, eddyz87, ihor.solodrai, linux-kselftest, ast, andrii,
daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]
> diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c
> index 684a8fb72414..ea6f1389245f 100644
> --- a/tools/bpf/bpftool/map.c
> +++ b/tools/bpf/bpftool/map.c
[ ... ]
> +/* Return 1 to use individual lookups, but only before batch traversal starts. */
> +static int dump_map_batch(int fd, void *key, void *value,
> + struct bpf_map_info *info, struct btf *btf,
> + json_writer_t *wtr, unsigned int *num_elems)
> +{
> + __u32 capacity, count, batch = 0, next_batch = 0, i;
> + void *keys = NULL, *values = NULL, *buf;
> + bool first = true, can_fallback = true;
> + int err;
[ ... ]
> + while (true) {
> + count = capacity;
> + err = bpf_map_lookup_batch(fd, first ? NULL : &batch,
> + &next_batch, keys, values, &count, NULL);
> + err = err ? errno : 0;
> + /*
> + * Older kernels reject the command before updating count. Do not
> + * inspect the buffers on these errors, or fall back after progress.
> + */
> + if (can_fallback && (err == EINVAL || err == EOPNOTSUPP ||
> + err == 524 /* ENOTSUPP */)) {
> + err = 1;
> + goto out;
> + }
> + can_fallback = false;
> + if (err == ENOSPC) {
> + if (capacity == info->max_entries)
> + goto error;
> + capacity += min(capacity, info->max_entries - capacity);
> + /* Preserve the input cursor: the oversized bucket was not read. */
> + goto resize;
> + }
> + /* In particular, EFAULT can leave count and the buffers invalid. */
> + if (err && err != ENOENT)
> + goto error;
[ ... ]
> + if (err == ENOENT) {
> + err = 0;
> + goto out;
> + }
> + first = false;
> + batch = next_batch;
> + }
> +error:
> + fprintf(stderr, "Error: can't lookup map batch: %s\n", strerror(err));
> + err = -1;
> +out:
> + free(keys);
> + free(values);
> + return err;
> +}
[ ... ]
> @@ -869,6 +965,9 @@ map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr,
> p_info("Warning: cannot read values from %s map with value_size != 8",
> map_type_str);
> }
> + err = dump_map_batch(fd, key, value, info, btf, wtr, &num_elems);
> + if (err != 1)
> + goto end_dump;
This isn't a bug, but would a separate flag for the fallback case read
more easily here? The err variable carries three different kinds of
values: negative returns from bpf_map_lookup_batch(), then positive
errno values after the "err = err ? errno : 0" line, and finally the
special value 1 to signal fallback to individual lookups. The sentinel
1 happens to equal EPERM numerically, though the logic does keep them
separate since all errno comparisons run before "err = 1" is assigned.
Could using something like "bool fall_back" or a small enum let err
stay purely an errno, so the caller's "if (err != 1)" doesn't have to
encode this third outcome?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34557203595
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-09-11 3:37 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-11 2:51 [PATCH bpf-next v3 0/2] bpftool: Batch bounded hash map dumps Tianyi Chen
2026-09-11 2:51 ` [PATCH bpf-next v3 1/2] bpftool: Use batch lookups for " Tianyi Chen
2026-09-11 3:37 ` bot+bpf-ci
2026-09-11 2:51 ` [PATCH bpf-next v3 2/2] selftests/bpf: Check bpftool batch map dump contents Tianyi Chen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox