From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 463192D6E72 for ; Sun, 7 Jun 2026 18:13:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780856016; cv=none; b=eFyvz343OFtkJrjRUkfeLX+YFESMjR/QTzBNvIfUsAKIuRlHaRPCIath2sRQlrqwvA2tgP0JptEQfQrYssnavdCBdujWleLPt9SlsAO1lf8Yo2Nm9Q0pvY3Hot80GaEUoePRkvBO2pALQw2diXkwhqG+rnK7Mi2iC7U3jla6iDY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780856016; c=relaxed/simple; bh=BP5TyjY1pCtGKAT//P7a7GvSHOPY7JKM31CFHBGd4Qs=; h=Mime-Version:Content-Type:Date:Message-Id:Subject:From:To:Cc: References:In-Reply-To; b=fKHTEKQasb/QYrxJ/+4DHpR9CXGQEIW8Xjtaq4DEALbn5/h/AOwarl4Q05L2wyJwsdMsOdgH3MB6VpdJYEOZ8Me4//2VX/NXMwxbai58YnqeLh5kDgcRVr531CGtD0gfwFdBo4QbSeVcjtmJfX3pAMmLFRbtjI7SFKRuQ1F3pGU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rE1QYYK0; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rE1QYYK0" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-486badc02ecso211550b6e.3 for ; Sun, 07 Jun 2026 11:13:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780856014; x=1781460814; darn=vger.kernel.org; h=in-reply-to:references:cc:to:from:subject:message-id:date :content-transfer-encoding:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=j3HXEYBCN+P/6hSxIy3mpYQoUowU8aTo9ESTr8/tzPI=; b=rE1QYYK0dVxik6mH8AsS/SFr8PiHyanCL93p7jT/6osWXHnTypGUVS3LqOklwLiyle 0mquJaZ6EAe9QMD9e08alKt85ELwTIb31vDgFQS6puIzyFVxCd3nIBu4Kxyt9Vqq/Oa6 WV3FKYDmcY9YsSJ0ZJCUy+cNHNuIBCMMMvNN7zUczkUIttU/6K8oyGeTkmzNcu/Ju8pe gWq+f/X/aQDqzvdwBmM3xNUtQxRRK/sYk70vCLG8gAcn1ESv/h1qDH1eauAbofGnOwDe v6Nzn3nywlK6HNY0tTHCM3B8C3qGkP6fyytmzKfaxEEGfzRVmBV1YG5g1Ok0PgSdSK+Y Mamw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780856014; x=1781460814; h=in-reply-to:references:cc:to:from:subject:message-id:date :content-transfer-encoding:mime-version:x-gm-gg:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=j3HXEYBCN+P/6hSxIy3mpYQoUowU8aTo9ESTr8/tzPI=; b=sW2B+hkx1smc8SbwjFK/0pCEmh5vUcZXsqkdecCklSLl+Sj49QcXV7/lzKJF19hkF0 BoNoviNcS11TSjWCHxnwRi4CfzNW8P2OAaaJYS0e8Juz1F6YPNwV2Lv5vTeYUds/aZLe DO6DJDD3XQGS0f7Z7vSRpBhP3CLaJcMSVLP5CTat3DchqOYc8KbvDIl1Clb1eoextyQ1 lAMgNvVmInhSCZXGucKYttmQwkLPn4lsTpxAIgFr8J81CRdXlYYNBgv7OnS+xRB799Tw pWIQNvUAL+tU017Z+iCK3j5P40htkuBoIB5tS7r7T4ssTXFBRJ3JZyyGRUoBvYfVX+lC 9LoA== X-Forwarded-Encrypted: i=1; AFNElJ9elvJPH8nGDsL4Zg/kuyb9mwYGNaHCSL3+8UUtMLOBKCMlbjZzLLU3zq+/4ucbz0XdtaAcrqpTLLjQvJH3o19YKJc=@vger.kernel.org X-Gm-Message-State: AOJu0YwMYbe1ahi30pDa08Ww46jt71FlWbW25IA0qjoqwEkhVA5xEYPf +EZMX21cYs18YVFXZ7vqbm+UMXC3FbLiO9+2v0bN+kC0hmiLFhxsS3cW X-Gm-Gg: Acq92OEHLx8593Uce3xAQiET1lwfnfxrLC2Ly+ABjgwQs5ypUTI4wq69PnKPy1Ms0/K sa5jqryDln7BIhBiMX8TDvwg5q8B6sJVsWQmoLw1RWYVIJ46yeNa64I3DJLvqNuDqxq1v4JHDCd wINhLokP9HD7NMJc+7zImUVxYMgIRwxabwMs3TJnFAoCOQj6D4x3OeDghGZfG1bkunlsWjDS332 zmxdSozVga91xtuW0cBIR5C1lCnjWOfeTVsFZYs6qAoUzrm/qLwFqcx9pZCFSjhVqV8diDgge5F vv75903yf5WoEc+1BTV2WMQs6hTlBmu85Xft2wjbl5SqURZnrj7+cS9H4c+t2ER1+/tRuwGOwSt DwmaYYfsEPC24IZUzXtgITwcZ+O5i+4CwB98UECQ4lBzMjgQvssr73SHSEYLa5o7+eJEQaPvRnK BzmzBOY2C4iskzgyFq3wM7mq/Ji8PXwp1cdqDR4nkC8WN+RekiFrGZSL9itWX0zsVemSHORd61g ab2RREmk+bnbqmPNNqaGWXn/I0= X-Received: by 2002:a05:6808:1a19:b0:486:52c6:36d8 with SMTP id 5614622812f47-4868db2dc14mr6846567b6e.5.1780856014039; Sun, 07 Jun 2026 11:13:34 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:8::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7e6e7965006sm10804134a34.23.2026.06.07.11.13.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 07 Jun 2026 11:13:33 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Sun, 07 Jun 2026 11:13:32 -0700 Message-Id: Subject: Re: [PATCHv8 bpf-next 28/29] selftests/bpf: Add tracing multi attach benchmark test From: "Alexei Starovoitov" To: "Jiri Olsa" , "Alexei Starovoitov" , "Daniel Borkmann" , "Andrii Nakryiko" Cc: , , "Martin KaFai Lau" , "Eduard Zingerman" , "Song Liu" , "Yonghong Song" , "Menglong Dong" , "Steven Rostedt" X-Mailer: aerc References: <20260606123955.345967-1-jolsa@kernel.org> <20260606123955.345967-29-jolsa@kernel.org> In-Reply-To: <20260606123955.345967-29-jolsa@kernel.org> On Sat Jun 6, 2026 at 5:39 AM PDT, Jiri Olsa wrote: > Adding benchmark test that attaches to (almost) all allowed tracing > functions and display attach/detach times. > > # ./test_progs -t tracing_multi_bench_attach -v > bpf_testmod.ko is already unloaded. > Loading bpf_testmod.ko... > Successfully loaded bpf_testmod.ko. > serial_test_tracing_multi_bench_attach:PASS:btf__load_vmlinux_btf 0 nse= c > serial_test_tracing_multi_bench_attach:PASS:tracing_multi_bench__open_a= nd_load 0 nsec > serial_test_tracing_multi_bench_attach:PASS:get_syms 0 nsec > serial_test_tracing_multi_bench_attach:PASS:bpf_program__attach_tracing= _multi 0 nsec > serial_test_tracing_multi_bench_attach: found 51186 functions > serial_test_tracing_multi_bench_attach: attached in 1.295s > serial_test_tracing_multi_bench_attach: detached in 0.243s ... > + if (!ASSERT_OK(bpf_get_ksyms(&ksyms, true), "get_syms")) > + goto cleanup; > + > + /* Get all ftrace 'safe' symbols.. */ > + for (i =3D 0; i < ksyms->filtered_cnt; i++) { > + if (!tsearch(&ksyms->filtered_syms[i], &root, compare)) { > + ASSERT_FAIL("tsearch failed"); > + goto cleanup; > + } > + } > + > + /* ..and filter them through BTF and btf_type_is_traceable_func. */ > + nr =3D btf__type_cnt(btf); > + for (type_id =3D 1; type_id < nr; type_id++) { > + const struct btf_type *type; > + const char *str; > + > + type =3D btf__type_by_id(btf, type_id); > + if (!type) > + break; > + > + if (BTF_INFO_KIND(type->info) !=3D BTF_KIND_FUNC) > + continue; > + > + str =3D btf__name_by_offset(btf, type->name_off); > + if (!str) > + break; > + > + if (!tfind(&str, &root, compare)) > + continue; > + > + if (!btf_type_is_traceable_func(btf, type)) > + continue; > + > + err =3D libbpf_ensure_mem((void **) &ids, &cap, sizeof(*ids), cnt + 1)= ; > + if (err) > + goto cleanup; > + > + ids[cnt++] =3D type_id; > + } This filtering wasn't enough. I've added removal of duplicates here while applying: + /* + * Collect names that are not unique in kallsyms. The kernel resolv= es a + * tracing-multi BTF id to an address with kallsyms_lookup_name(), = which + * returns the first symbol of that name. For a duplicate name that= may + * be a different (non-ftrace-able) instance than the ftrace-able o= ne in + * available_filter_functions, so attaching to it by BTF id fails w= ith + * -ENOENT (e.g. t_start/t_next/t_stop). ksyms->syms is sorted by n= ame, + * so equal names are adjacent. + */ + for (i =3D 1; i < ksyms->sym_cnt; i++) { + if (strcmp(ksyms->syms[i].name, ksyms->syms[i - 1].name)) + continue; + if (!tsearch(&ksyms->syms[i].name, &dups, compare)) { + ASSERT_FAIL("tsearch failed"); + goto cleanup; + } + } + /* Skip names that are not unique in kallsyms, see above. *= / + if (tfind(&str, &dups, compare)) + continue; As claude explains it: ---- 1. The kernel attaches tracing_multi by BTF id. To get an address it resolv= es the BTF function name via kallsyms_lookup_name(tname) and requires ftrace_location(addr) =E2=80=94 kernel/bpf/verifier.c:19380: addr =3D kallsyms_lookup_name(tname); ... if (!addr || !ftrace_location(addr)) return -ENOENT; 2. t_start/t_next/t_stop each have 5 instances in this kernel. Only one is ftrace-able =E2=80=94 the copies in kernel/trace/* are built notrace (ftrac= e's Makefile strips -pg), so only the unrelated copy is in available_filter_functions: 3. kallsyms_lookup_name() returns the lowest-address instance among equal n= ames (exact strcmp, lowest seq). That instance has no fentry =E2=86=92 ftrace_lo= cation() returns 0 =E2=86=92 -ENOENT, which aborts the whole all-or-nothing bench at= tach. Why the bench includes them: it intersects BTF FUNC names with available_filter_functions names. Since some t_start is ftrace-able, the na= me passes the filter =E2=80=94 but the kernel resolves the wrong (non-ftrace-a= ble) t_start. The author's kernel apparently had the ftrace-able copy at the low= est address, so it passed there. This is a pre-existing limitation, not multi-specific: single fentry attach= by BTF id uses the same kallsyms_lookup_name(tname) path (verifier.c:19120) = =E2=80=94 you can't reliably fentry-attach to any duplicate-named function on this kernel either. ---- Maybe we should adjust bpf_get_ksyms() instead. Not sure. After the fix in my gcc 15 kernel with KASAN: serial_test_tracing_multi_bench_attach: found 49045 functions serial_test_tracing_multi_bench_attach: attached in 4.115s serial_test_tracing_multi_bench_attach: detached in 2.313s