Re: [External] Re: [PATCH bpf-next v2 1/9] bpf: tracing: add support to record and check the accessed args

Andrii Nakryiko
2024-03-31T16:37:19Z

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 45494 bytes --]

On Thu, Mar 28, 2024 at 8:10 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 28 Mar 2024 22:43:46 +0800
> æ¢¦é¾™è‘£ <dongmenglong.8@bytedance.com> wrote:
>
> > I have done a simple benchmark on creating 1000
> > trampolines. It is slow, quite slow, which consume up to
> > 60s. We can't do it this way.
> >
> > Now, I have a bad idea. How about we introduce
> > a "dynamic trampoline"? The basic logic of it can be:
> >
> > """
> > save regs
> > bpfs = trampoline_lookup_ip(ip)
> > fentry = bpfs->fentries
> > while fentry:
> > fentry(ctx)
> > fentry = fentry->next
> >
> > call origin
> > save return value
> >
> > fexit = bpfs->fexits
> > while fexit:
> > fexit(ctx)
> > fexit = fexit->next
> >
> > xxxxxx
> > """
> >
> > And we lookup the "bpfs" by the function ip in a hash map
> > in trampoline_lookup_ip. The type of "bpfs" is:
> >
> > struct bpf_array {
> > struct bpf_prog *fentries;
> > struct bpf_prog *fexits;
> > struct bpf_prog *modify_returns;
> > }
> >
> > When we need to attach the bpf progA to function A/B/C,
> > we only need to create the bpf_arrayA, bpf_arrayB, bpf_arrayC
> > and add the progA to them, and insert them to the hash map
> > "direct_call_bpfs", and attach the "dynamic trampoline" to
> > A/B/C. If bpf_arrayA exist, just add progA to the tail of
> > bpf_arrayA->fentries. When we need to attach progB to
> > B/C, just add progB to bpf_arrayB->fentries and
> > bpf_arrayB->fentries.
> >
> > Compared to the trampoline, extra overhead is introduced
> > by the hash lookuping.
> >
> > I have not begun to code yet, and I am not sure the overhead is
> > acceptable. Considering that we also need to do hash lookup
> > by the function in kprobe_multi, maybe the overhead is
> > acceptable?
>
> Sounds like you are just recreating the function management that ftrace
> has. It also can add thousands of trampolines very quickly, because it does
> it in batches. It takes special synchronization steps to attach to fentry.
> ftrace (and I believe multi-kprobes) updates all the attachments for each
> step, so the synchronization needed is only done once.
>
> If you really want to have thousands of functions, why not just register it
> with ftrace itself. It will give you the arguments via the ftrace_regs
> structure. Can't you just register a program as the callback?
>
> It will probably make your accounting much easier, and just let ftrace
> handle the fentry logic. That's what it was made to do.
>

I thought I'll just ask instead of digging through code, sorry for
being lazy :) Is there any way to pass pt_regs/ftrace_regs captured
before function execution to a return probe (fexit/kretprobe)? I.e.,
how hard is it to pass input function arguments to a kretprobe? That's
the biggest advantage of fexit over kretprobe, and if we can make
these original pt_regs/ftrace_regs available to kretprobe, then
multi-kretprobe will effectively be this multi-fexit.

> -- Steve

X-sender: <netdev+bounces-83483-steffen.klassert=cunet.com@vger.kernel.org>
X-Receiver: <steffen.klassert@secunet.com> ORCPT=c822;steffen.klassert@secunet.com NOTIFY=VER; X-ExtendedProps=AVABYAAgAAAAUAFAARAPDFCS25BAlDktII2g02frgPADUAAABNaWNyb3NvZnQuRXhjaGFuZ2UuVHJhbnNwb3J0LkRpcmVjdG9yeURhdGEuSXNSZXNvdXJjZQIAAAUAagAJAAEAAAAAAAAABQAWAAIAAAUAQwACAAAFAEYABwADAAAABQBHAAIAAAUAEgAPAGIAAAAvbz1zZWN1bmV0L291PUV4Y2hhbmdlIEFkbWluaXN0cmF0aXZlIEdyb3VwIChGWURJQk9IRjIzU1BETFQpL2NuPVJlY2lwaWVudHMvY249U3RlZmZlbiBLbGFzc2VydDY4YwUACwAXAL4AAACheZxkHSGBRqAcAp3ukbifQ049REI2LENOPURhdGFiYXNlcyxDTj1FeGNoYW5nZSBBZG1pbmlzdHJhdGl2ZSBHcm91cCAoRllESUJPSEYyM1NQRExUKSxDTj1BZG1pbmlzdHJhdGl2ZSBHcm91cHMsQ049c2VjdW5ldCxDTj1NaWNyb3NvZnQgRXhjaGFuZ2UsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1zZWN1bmV0LERDPWRlBQAOABEABiAS9uuMOkqzwmEZDvWNNQUAHQAPAAwAAABtYngtZXNzZW4tMDIFADwAAgAADwA2AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50LkRpc3BsYXlOYW1lDwARAAAAS2xhc3NlcnQsIFN0ZWZmZW4FAAwAAgAABQBsAAIAAAUAWAAXAEoAAADwxQktuQQJQ5LSCNoNNn64Q049S2xhc3NlcnQgU3RlZmZlbixPVT1Vc2VycyxPVT1NaWdyYXRpb24sREM9c2VjdW5ldCxEQz1kZQUAJgACAAEFACIADwAxAAAAQXV0b1Jlc3BvbnNlU3VwcHJlc3M6IDANClRyYW5zbWl0SGlzdG9yeTogRmFsc2UNCg8ALwAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuRXhwYW5zaW9uR3JvdXBUeXBlDwAVAAAATWVtYmVyc0dyb3VwRXhwYW5zaW9uBQAjAAIAAQ=
X-CreatedBy: MSExchange15
X-HeloDomain: a.mx.secunet.com
X-ExtendedProps: BQBjAAoAo5Tp8x1Q3AgFAGEACAABAAAABQA3AAIAAA8APAAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuTWFpbFJlY2lwaWVudC5Pcmdhbml6YXRpb25TY29wZREAAAAAAAAAAAAAAAAAAAAAAAUASQACAAEFAGIACgBuAAAAjYoAAAUABAAUIAEAAAAcAAAAc3RlZmZlbi5rbGFzc2VydEBzZWN1bmV0LmNvbQUABgACAAEFACkAAgABDwAJAAAAQ0lBdWRpdGVkAgABBQACAAcAAQAAAAUAAwAHAAAAAAAFAAUAAgABBQBkAA8AAwAAAEh1Yg=
X-Source: SMTP:Default MBX-DRESDEN-01
X-SourceIPAddress: 62.96.220.36
X-EndOfInjectedXHeaders: 19722
Received: from cas-essen-02.secunet.de (10.53.40.202) by
mbx-dresden-01.secunet.de (10.53.40.199) with Microsoft SMTP Server
(version=S1_2, cipher=S_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2507.37; Sat, 30 Mar 2024 00:29:03 +0100
Received: from a.mx.secunet.com (62.96.220.36) by cas-essen-02.secunet.de
(10.53.40.202) with Microsoft SMTP Server (version=S1_2,
cipher=S_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend
Transport; Sat, 30 Mar 2024 00:29:03 +0100
Received: from localhost (localhost [127.0.0.1])
by a.mx.secunet.com (Postfix) with ESMTP id D07E8208A6
for <steffen.klassert@secunet.com>; Sat, 30 Mar 2024 00:29:03 +0100 (CET)
X-Virus-Scanned: by secunet
X-Spam-Flag: NO
X-Spam-Score: -5.049
X-Spam-Level:
X-Spam-Status: No, score=.049 tagged_above=99 required=1
tests=AYES_00=.9, DKIM_SIGNED=1, DKIM_VALID=.1,
DKIM_VALID_AU=.1, FREEMAIL_FORGED_FROMDOMAIN=001,
FREEMAIL_FROM=001, HEADER_FROM_DIFFERENT_DOMAINS=249,
MAILING_LIST_MULTI=, RCVD_IN_DNSWL_MED=.3, SPF_HELO_NONE=001,
SPF_PASS=.001] autolearn=m autolearn_force=
Authentication-Results: a.mx.secunet.com (amavisd-new);
dkim=ss (2048-bit key) header.d=ail.com
Received: from a.mx.secunet.com ([127.0.0.1])
by localhost (a.mx.secunet.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id Ex4lJg4QxVwR for <steffen.klassert@secunet.com>;
Sat, 30 Mar 2024 00:29:02 +0100 (CET)
Received-SPF: Pass (sender SPF authorized) identity=ilfrom; client-ip\x139.178.88.99; helo=.mirrors.kernel.org; envelope-from=tdev+bounces-83483-steffen.klassert=cunet.com@vger.kernel.org; receiver=effen.klassert@secunet.com
DKIM-Filter: OpenDKIM Filter v2.11.0 a.mx.secunet.com 1CA0120870
Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org [139.178.88.99])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by a.mx.secunet.com (Postfix) with ESMTPS id 1CA0120870
for <steffen.klassert@secunet.com>; Sat, 30 Mar 2024 00:29:00 +0100 (CET)
Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by sv.mirrors.kernel.org (Postfix) with ESMTPS id E018C2831AB
for <steffen.klassert@secunet.com>; Fri, 29 Mar 2024 23:28:58 +0000 (UTC)
Received: from localhost.localdomain (localhost.localdomain [127.0.0.1])
by smtp.subspace.kernel.org (Postfix) with ESMTP id 1120F13E40E;
Fri, 29 Mar 2024 23:28:49 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org;
dkim=ss (2048-bit key) header.d=ail.com header.i=mail.com header.b=m8qb98v"
X-Original-To: netdev@vger.kernel.org
Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169])
(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
(No client certificate requested)
by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEAE5347B6;
Fri, 29 Mar 2024 23:28:46 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=ne smtp.client-ip 9.85.215.169
ARC-Seal: i= a=a-sha256; d=bspace.kernel.org; s=c-20240116;
t\x1711754928; cv=ne; b=F0sOPYXhrbjY9cOS6Nmj4VXF++Yx8MtZjBDxanMefli+BulHO4QXKn7qwpW3i+7uN65u5cLGhThPJZ2akcLjIgY9UcOJfH9j6D484gs+N5O8HXdnQzg8eZ93GRBeBoDesuvP5WyVY1RlteRQZLbYRhS27z9mKgIzxIt2bJ+ZoARC-Message-Signature: i= a=a-sha256; d=bspace.kernel.org;
s=c-20240116; t\x1711754928; c=laxed/simple;
bh=CkTC/9QOPKjcAq26pJFK75aD0g4y2+LXJsB0P1JY0=
h=ME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:
To:Cc:Content-Type; b=4xUrtWjk3J0kbVbPD5XAlV9JEyhRuyqvveB8mIxkffhtOM1Y7m4+o5XzftYMKn592GD5CBw41tS2CCKYzMtaJurKzqjxkWsFCCe3ELmR9AEyVvEBDa4PpVwvE2aS0et1z2P9rOIk0ldc7OMYPnDpWEu3jfzLxb8Y4CGi6sJbEARC-Authentication-Results: i= smtp.subspace.kernel.org; dmarc=ss (p=ne dis=ne) header.from=ail.com; spf=ss smtp.mailfrom=ail.com; dkim=ss (2048-bit key) header.d=ail.com header.i=mail.com header.b=8qb98v; arc=ne smtp.client-ip 9.85.215.169
Authentication-Results: smtp.subspace.kernel.org; dmarc=ss (p=ne dis=ne) header.from=ail.com
Authentication-Results: smtp.subspace.kernel.org; spf=ss smtp.mailfrom=ail.com
Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5d3907ff128so1860607a12.3;
Fri, 29 Mar 2024 16:28:46 -0700 (PDT)
DKIM-Signature: v= a=a-sha256; c=laxed/relaxed;
d=ail.com; s 230601; t\x1711754926; x\x1712359726; darn=er.kernel.org;
h=ntent-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:from:to:cc:subject:date
:message-id:reply-to;
bh=V1tZGanDBSafK6edOdycvbQYDEdHbTCxngJvsvk3c=
b=8qb98vO+2rZQDr4ACVbzWOOBhMbNP973of56VDx+7dzKarS23b4VDs0QCZjE/zey
IJ/g2gwkTlDZp557gqFy6HVirUKvvI/RhzQOrPjsYmqnt0OS+LXZWxBGcD9ooJmS1u1N
Wdq60dFcb/ee3JYL5bUFYzr07cJmwvvmIZoI/b1KQpo+K6ftMzexzXAN8iXh1nJo1QdC
4CFMaxfeoUIzc6fyg2ScGW/5rPdZFOXPIXIjGrA7V7Ij1KqAkHvKfDNzy1QS7jMz6z0E
/W8l0QAzZuWpon/+MGFe6iPNG81xSsUfka4lIyvOK31ZY63E/sqixJRnh7RpDV18WsJY
CtMw=
X-Google-DKIM-Signature: v= a=a-sha256; c=laxed/relaxed;
d\x1e100.net; s 230601; t\x1711754926; x\x1712359726;
h=ntent-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=V1tZGanDBSafK6edOdycvbQYDEdHbTCxngJvsvk3c=
b=6CYGdij3I8SMDFuLcEkXFxxuMDLhpfo6RmyHf+rD3z+RBxrsmoJ1YligUcYT3VoL
YtUHguMvFF7xT9YjyGrK4ujkEA13NT+X1hwUNhKsWaXhpLBG2Wyh4eNaw4zkHN28eYkq
dj4reqkSxtXQvjpns3NtubA0ld6+woHqICxMQVR26maDt91aYEPooASIjOoCRnxMnIpI
KqBMby8F8aarmJfc6c16rp9hDJK1Cr34usuvnIOrnJ2bdp2qTCOMpl6ARzTAtPA9p7n7
9p9R345kssoRHOmDzo2WcF/0oG4wRjb+owNVpsCTExoqfee0d+FSjcyFxitgRQvMQzcC
9Pqw=
X-Forwarded-Encrypted: i= AJvYcCVlnHyNUDZ0fbOQSV0NEVvOtJTaOmaxwPgZy047hqztaIz7SMsnLeomStI1EizImJzlky/xJNYIS4RJsXEX2rbQoVyo7MFG2+RfwxWnjGonfBHU0k5JSlRGPFtF8Te8neu3qxCi25AWVXDje1k1qkzJLjDnQ+s4R4WJmDmbw/KdNN4l/EBnMx0Wq+GOvmC+DkjFlyebAZoKGfgwdw8BgI1xISF1T+wxXijKToVLbwSSua486L4tXm8nE4lXExEMG+QxE02AxDYDKshvWmyLqPiNewg+cXpbihxLyg=
X-Gm-Message-State: AOJu0YwCezoL4rml44Ulwt5kaSYk47W3cr0evd5PMAYC8La4m9gnILHN
FRelRR6zMz1Bz2qJo3eFKeNyq28N5Yo0kJIQ72rv0EkZFGKfSnjbac0fnT9zQAwbNl/tFnTmfuG
Y3REKAMgIbWmVvmHWWGOk0v3R69UX-Google-Smtp-Source: AGHT+IFNbTnA2C1UowyvpIWfWMecar3RYOVPJRlOoVy5NMmPOOKB6ZhvqfzCxbpKkf/w3Z3THbX/GA+A1iwjf07EvPYX-Received: by 2002:a17:90a:be04:b0:2a2:1415:723d with SMTP id
a4-20020a17090abe0400b002a21415723dmr3177860pjs.42.1711754926271; Fri, 29 Mar
2024 16:28:46 -0700 (PDT)
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com>
<20240311093526.1010158-2-dongmenglong.8@bytedance.com> <CAADnVQKQPS5NcvEouH4JqZ2fKgQAC+LtcwhX9iXYoiEkF_M94Q@mail.gmail.com>
<CALz3k9i5G5wWi+rtvHPwVLOUAXVMCiU_8QUZs87TEYgR_0wpPA@mail.gmail.com>
<CAADnVQJ_ZCzMmT1aBsNXEBFfYNSVBdBXmLocjR0PPEWtYQrQFw@mail.gmail.com>
<CALz3k9icPePb0c4FE67q=U0hrePorN9gDpQrKTR_sXbLMfDA@mail.gmail.com>
<CAADnVQLwgw8bQ7OHBbqLhcPJ2QpxiGw3fkMFur+2cjZpM_78oA@mail.gmail.com>
<CALz3k9g9k7fEwdTZVLhrmGoXp8CE47Q+83r-AZDXrzzuR+CjVA@mail.gmail.com>
<CAADnVQLHpi3J6cBJ0QBgCQ2aY6fWGnVvNGdfi3W-jmoa9d1eVQ@mail.gmail.com>
<CALz3k9g-U8ih=JPRbyU9x_9cp00fNkU3PGQ6jP0WJ+=mqQ@mail.gmail.com>
<CALz3k9jG5Jrqw=jt05yMkEF-1u909GbBYrV-02W0dQtm6KQQ@mail.gmail.com> <20240328111330.194dcbe5@gandalf.local.home>
In-Reply-To: <20240328111330.194dcbe5@gandalf.local.home>
From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Date: Fri, 29 Mar 2024 16:28:33 -0700
Message-ID: <CAEf4BzYgzOti+Hfdn3SUCjuofGedXRSGApVDD+K2TdG6oNE-pw@mail.gmail.com>
Subject: Re: [External] Re: [PATCH bpf-next v2 1/9] bpf: tracing: add support
to record and check the accessed args
To: Steven Rostedt <rostedt@goodmis.org>
Cc: =TF-8?B?5qKm6b6Z6JGj?=ongmenglong.8@bytedance.com>,
Alexei Starovoitov <alexei.starovoitov@gmail.com>, Jiri Olsa <jolsa@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>, Martin KaFai Lau <martin.lau@linux.dev>, Eddy Z <eddyz87@gmail.com>,
Song Liu <song@kernel.org>, Yonghong Song <yonghong.song@linux.dev>,
John Fastabend <john.fastabend@gmail.com>, KP Singh <kpsingh@kernel.org>,
Stanislav Fomichev <sdf@google.com>, Hao Luo <haoluo@google.com>,
Alexander Gordeev <agordeev@linux.ibm.com>, Christian Borntraeger <borntraeger@linux.ibm.com>,
Sven Schnelle <svens@linux.ibm.com>, "David S. Miller" <davem@davemloft.net>,
David Ahern <dsahern@kernel.org>, Dave Hansen <dave.hansen@linux.intel.com>,
X86 ML <x86@kernel.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Quentin Monnet <quentin@isovalent.com>, bpf <bpf@vger.kernel.org>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>, LKML <linux-kernel@vger.kernel.org>,
linux-riscv <linux-riscv@lists.infradead.org>, linux-s390 <linux-s390@vger.kernel.org>,
Network Development <netdev@vger.kernel.org>, linux-trace-kernel@vger.kernel.org,
"open list:KERNEL SELFTEST FRAMEWORK" <linux-kselftest@vger.kernel.org>, linux-stm32@st-md-mailman.stormreply.com
Content-Type: text/plain; charset=TF-8"
Content-Transfer-Encoding: quoted-printable
Return-Path: netdev+bounces-83483-steffen.klassert=cunet.com@vger.kernel.org
X-MS-Exchange-Organization-OriginalArrivalTime: 29 Mar 2024 23:29:03.8846
(UTC)
X-MS-Exchange-Organization-Network-Message-Id: d7c7485a-e2b7-4948-127e-08dc50480570
X-MS-Exchange-Organization-OriginalClientIPAddress: 62.96.220.36
X-MS-Exchange-Organization-OriginalServerIPAddress: 10.53.40.202
X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: cas-essen-02.secunet.de
X-MS-Exchange-Organization-OrderedPrecisionLatencyInProgress: LSRV=x-dresden-01.secunet.de:TOTAL-HUB=432|SMR=337(SMRDE=034|SMRC=302(SMRCL=102|X-SMRCR=303))|CAT=093(CATOS=011
(CATSM=011(CATSM-Malware
Agent=011))|CATRESL=039(CATRESLP2R=020)|CATORES=041
(CATRS=041(CATRS-Index Routing Agent=040)));2024-03-29T23:29:04.328Z
X-MS-Exchange-Forest-ArrivalHubServer: mbx-dresden-01.secunet.de
X-MS-Exchange-Organization-AuthSource: cas-essen-02.secunet.de
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-FromEntityHeader: Internet
X-MS-Exchange-Organization-OriginalSize: 13304
X-MS-Exchange-Organization-HygienePolicy: Standard
X-MS-Exchange-Organization-MessageLatency: SRVÊs-essen-02.secunet.de:TOTAL-FE=012|SMR=009(SMRPI=007(SMRPI-FrontendProxyAgent=007))|SMS=002
X-MS-Exchange-Organization-AVStamp-Enterprise: 1.0
X-MS-Exchange-Organization-Recipient-Limit-Verified: True
X-MS-Exchange-Organization-TotalRecipientCount: 1
X-MS-Exchange-Organization-Rules-Execution-History: 0b0cf904-14ac-4724-8bdf-482ee6223cf2%%%fd34672d-751c-45ae-a963-ed177fcabe23%%%d8080257-b0c3-47b4-b0db-23bc0c8ddb3c%%%95e591a2-5d7d-4afa-b1d0-7573d6c0a5d9%%%f7d0f6bc-4dcc-4876-8c5d-b3d6ddbb3d55%%%16355082-c50b-4214-9c7d-d39575f9f79b
X-MS-Exchange-Forest-RulesExecuted: mbx-dresden-01
X-MS-Exchange-Organization-RulesExecuted: mbx-dresden-01
X-MS-Exchange-Forest-IndexAgent-0: AQ0CZW4AAfkHAAAPAAADH4sIAAAAAAAEAIVWzW7kxhHu0cxw/qQdIb
ZhwMihocuuktGsrCwMQXa01u4lOjgB4gA+Cj1kzwwjDsmQTY3oIEBe
IcdcEyDXvEHueY0cDPst/FUVyeFo14hAjLqbVV9VffXT/O/Hv4v1H9
bFTH9lMn1xOdMX5xevtHH68urT8//99d83X+mvnX2wsf59kjsbOP1F
JosvV0kSbMJ8nmSra73NEmevjsbXeHSNeXEpsAR5cXH16ldXrz7Tvz
y/PD8nqe/+8a8f/vP37//2T/1FkMSrjY1XEf7PL79clDBgYt/O/WTz
BPta3+q1ebAaOlYbnYebNLJ6YWN/vTHZvU5i7WfWuDBe6U/PxdS1dp
nZpEkUxjaf61unw1znUbKd6T8VobPVersO/bX2kzgvNlYXqXaJaH92
Dq1vrPZN/NzBsg6ddmtgbE05ZxGR+y2hVP4ZvTCBDgNr5vo3yVabRV
I4vbU6jF2WBIVvRcfok6CMzSb0W06evAaDiMrkOI6SFX6TJVmFB4j1
qmXz5OREFjlZzewql+0iXeb61y3MuyhJ7ov0LkxfhOmpCC0tnCkhRt
Jn17wNbYUAOsCsiIhFravtC989nu4fAUQWZ9exfXQtD30TRTrJwlUY
73nqiizWDyYqbEt4aR8R5s4h7J64g5OWN9g9cUb0+f87rjzy3z5xsr
uJA0qOcITkWn1CLpzoRcm7ZRH7LkRxhSkyiKytTb7WG5OKOo7ex/Sc
0+jK1FL+KsQwb+cvd1nhO4r3zmSZKfWf61Bab9IsWelf1On5vJJ4jw
DR9flPAmySIFyWd0J9LfeXljPfrNHpoCG2NkD1YxA4g5YgAoCiCeWG
zhs2bl6+efl2ViUIQcZR2ShzG9paWaK7me3Wb1rrt1UzIAsmCFinMY
bNZsavwji3meOD6sWTNJwEYWZ9d0cld8dsi2Irjvd1W9PoHA4GxLLl
sQapuZvpPxa5Y+fajmlnQhT3sum5SmnXS/OfIpVg3jSWicZ9E/SuRd
YOkSJ6Yq/1tj2P3iab1GRilZ1tQp4hKux08mCztaVBle8GU41e7hiW
isZMbcNXky5OUGN2VcSc9CSwurROeL/VZsPv8yKTSmgZrFLu+zZ1Zh
HZOfyNc0zMjGa3WxselybKk4Y6jN6WO3tu7ho01vfgb2HvNkXkwhmq
o1z8X+Ovq/vl66SIg1xH4T3CSAoN+iQtqKv6WtkztzGxWVncXk5cXo
JWme3wlO8ajoAGt1R2UuSGTGAgtG4lDddKuo38+6hEY1jfFLmlkR8k
MpCxRGgL4/x1dYk5cw/FPLV+aCKdl7gCsyQOvzXsGG7pNG83cVINaE
6huKlfSJYWNgpxy2tm7Ezoy09xBQboYBQcBjgFLUgUa66XSaYtdgRG
pmY6lyJ76gflznKB8XTgezuhy71iHM1GRIPcCK+3hohMpLL2yKoZz+
miLrmqqsSs0J82A0EEtw3duo4Ow9BGS+ZqGyKGVfggWeVgslUhoTyE
RnLKWnf1HSrjE4WLwuSLnzT3TRruVGRRm5whaPAsjH9fl1NtmRhFmZ
UoFymsjGoPtcYFtSmQH4vL3mbSN2wlsvvVFAeRlHF14fJ3Ad0wxj3H
pwgVH2pkC082Bk3I7SIkH41vmcvV2unb53BHBk1+TxPVUUOA3yBcra
S6M5LkTqakZjCFZB+NF5ZeR+bbUl+d6lsOGN1hYspaSfZSk+c6dczg
yxaboCUlIgMCAVarfeyj9QteUaXWXwVcgPoF32cv73HIB6ev9e3czn
HdrPFBtTaZjC3XmA7jFN9YDfYuw4zd4LyuODsa892EuC3P3QfUHnqZ
i40/I2he7NSqO2hJU4namVLJEOhT+bxBF74vevOAS4KGDPnRgoMqvo
iqnquPpV7scol7DOWKkuHhhUBFkD2bU06v9dmZfJ3TTqkD1e2qfrej
DvGo3oHq9VT/oNMdKDVQQ08NBmo06KgP1BBbSPbVkagcdNQED617vU
5/qNSwM+or1VcjvAXaz9VRRx2M1Ji1plgDGQ8WgMU5cKDSU6qnxhOG
xdNTXl958guLfEh2scBJLdPvqmmXoQSQfFAefkVmROYQwnCsjuSw0e
1SjB+zJ+OeGow76kM1gT/8duQp+OMN1TPIe2qK86469Oh82FNTaHnq
kH8HfXXcp0OIHTI+qXRpPeyrQY90P2FbXQaBOU+sD9VIPBHnu7xtud
HrSgo4ogP1jNQ7akRRTGU95FdDddzeevtvvY56Rrn7hKG68sue9BqW
4Al84GcoMntiHTVWPQT+zuJQQBocYd6jqPsSI8c1hg/HfNhjQOGBk3
KInNamOSkV8oCp6zfINRU7H4aUtb2IWFLQunu6OxUU5Ki1/VBSAMkx
5062Y3XcWBypj+rsNDH2u7zFeStGySZFNOHCaNMoa4+C3eXdoy6jML
laGvWfeXXggjxWE3mLBYOMuzW4yEj7iFHq0CqD0i+o/JH0C86nVKie
cMW/E0olB4K3aJ4jEhtxdxPyhPPb49oWz6dcaU1ovTqiJuPM2Jj7bt
KEwKF50ixeq8Aa/2Xh0QwBzhC6wonoins1dceiPqT27As/jDxsEgGK
vLrM+mRxrxqZlrFgNm4I/oAnmwwcnlRekz7xR7qYpyJGIue08hn+kF
F4JeHLZJPkjojGQct6VQ9Sb438qE7HgBgYi1HmcyJ26xC8qlDf8aFh
nmQ6akqmqUcmnCaySJJQn/Z/BIV/1au5EQAAAQLeAjw/eG1sIHZlcn
Npb249IjEuMCIgZW5jb2Rpbmc9InV0Zi0xNiI/Pg0KPFRhc2tTZXQ+
DQogIDxWZXJzaW9uPjE1LjAuMC4wPC9WZXJzaW9uPg0KICA8VGFza3
M+DQogICAgPFRhc2sgU3RhcnRJbmRleD0iMTY2OSI+DQogICAgICA8
VGFza1N0cmluZz5Db25zaWRlcmluZyB0aGF0IHdlIGFsc28gbmVlZC
B0byBkbyBoYXNoIGxvb2t1cDwvVGFza1N0cmluZz4NCiAgICAgIDxB
c3NpZ25lZXM+DQogICAgICAgIDxFbWFpbFVzZXIgSWQ9InJvc3RlZH
RAZ29vZG1pcy5vcmciPlN0ZXZlbiBSb3N0ZWR0PC9FbWFpbFVzZXI+
DQogICAgICA8L0Fzc2lnbmVlcz4NCiAgICA8L1Rhc2s+DQogIDwvVG
Fza3M+DQo8L1Rhc2tTZXQ+AQrEAjw/eG1sIHZlcnNpb249IjEuMCIg
ZW5jb2Rpbmc9InV0Zi0xNiI/Pg0KPEVtYWlsU2V0Pg0KICA8VmVyc2
lvbj4xNS4wLjAuMDwvVmVyc2lvbj4NCiAgPEVtYWlscz4NCiAgICA8
RW1haWwgU3RhcnRJbmRleD0iNDgiPg0KICAgICAgPEVtYWlsU3RyaW
5nPnJvc3RlZHRAZ29vZG1pcy5vcmc8L0VtYWlsU3RyaW5nPg0KICAg
IDwvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0SW5kZXg9IjEyNSI+DQ
ogICAgICA8RW1haWxTdHJpbmc+ZG9uZ21lbmdsb25nLjhAYnl0ZWRh
bmNlLmNvbTwvRW1haWxTdHJpbmc+DQogICAgPC9FbWFpbD4NCiAgPC
9FbWFpbHM+DQo8L0VtYWlsU2V0PgEM8AM8P3htbCB2ZXJzaW9uPSIx
LjAiIGVuY29kaW5nPSJ1dGYtMTYiPz4NCjxDb250YWN0U2V0Pg0KIC
A8VmVyc2lvbj4xNS4wLjAuMDwvVmVyc2lvbj4NCiAgPENvbnRhY3Rz
Pg0KICAgIDxDb250YWN0IFN0YXJ0SW5kZXg9IjMyIj4NCiAgICAgID
xQZXJzb24gU3RhcnRJbmRleD0iMzIiPg0KICAgICAgICA8UGVyc29u
U3RyaW5nPlN0ZXZlbiBSb3N0ZWR0PC9QZXJzb25TdHJpbmc+DQogIC
AgICA8L1BlcnNvbj4NCiAgICAgIDxFbWFpbHM+DQogICAgICAgIDxF
bWFpbCBTdGFydEluZGV4PSI0OCI+DQogICAgICAgICAgPEVtYWlsU3
RyaW5nPnJvc3RlZHRAZ29vZG1pcy5vcmc8L0VtYWlsU3RyaW5nPg0K
ICAgICAgICA8L0VtYWlsPg0KICAgICAgPC9FbWFpbHM+DQogICAgIC
A8Q29udGFjdFN0cmluZz5TdGV2ZW4gUm9zdGVkdCAmbHQ7cm9zdGVk
dEBnb29kbWlzLm9yZzwvQ29udGFjdFN0cmluZz4NCiAgICA8L0Nvbn
RhY3Q+DQogIDwvQ29udGFjdHM+DQo8L0NvbnRhY3RTZXQ+AQ7PAVJl
dHJpZXZlck9wZXJhdG9yLDEwLDE7UmV0cmlldmVyT3BlcmF0b3IsMT
EsMTtQb3N0RG9jUGFyc2VyT3BlcmF0b3IsMTAsMDtQb3N0RG9jUGFy
c2VyT3BlcmF0b3IsMTEsMDtQb3N0V29yZEJyZWFrZXJEaWFnbm9zdG
ljT3BlcmF0b3IsMTAsMTtQb3N0V29yZEJyZWFrZXJEaWFnbm9zdGlj
T3BlcmF0b3IsMTEsMDtUcmFuc3BvcnRXcml0ZXJQcm9kdWNlciwyMC wyOA=
X-MS-Exchange-Forest-IndexAgent: 1 3445
X-MS-Exchange-Forest-EmailMessageHash: 272706DD
X-MS-Exchange-Forest-Language: en
X-MS-Exchange-Organization-Processed-By-Journaling: Journal Agent

On Thu, Mar 28, 2024 at 8:10 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 28 Mar 2024 22:43:46 +0800
> æ¢¦é¾™è‘£ <dongmenglong.8@bytedance.com> wrote:
>
> > I have done a simple benchmark on creating 1000
> > trampolines. It is slow, quite slow, which consume up to
> > 60s. We can't do it this way.
> >
> > Now, I have a bad idea. How about we introduce
> > a "dynamic trampoline"? The basic logic of it can be:
> >
> > """
> > save regs
> > bpfs = trampoline_lookup_ip(ip)
> > fentry = bpfs->fentries
> > while fentry:
> > fentry(ctx)
> > fentry = fentry->next
> >
> > call origin
> > save return value
> >
> > fexit = bpfs->fexits
> > while fexit:
> > fexit(ctx)
> > fexit = fexit->next
> >
> > xxxxxx
> > """
> >
> > And we lookup the "bpfs" by the function ip in a hash map
> > in trampoline_lookup_ip. The type of "bpfs" is:
> >
> > struct bpf_array {
> > struct bpf_prog *fentries;
> > struct bpf_prog *fexits;
> > struct bpf_prog *modify_returns;
> > }
> >
> > When we need to attach the bpf progA to function A/B/C,
> > we only need to create the bpf_arrayA, bpf_arrayB, bpf_arrayC
> > and add the progA to them, and insert them to the hash map
> > "direct_call_bpfs", and attach the "dynamic trampoline" to
> > A/B/C. If bpf_arrayA exist, just add progA to the tail of
> > bpf_arrayA->fentries. When we need to attach progB to
> > B/C, just add progB to bpf_arrayB->fentries and
> > bpf_arrayB->fentries.
> >
> > Compared to the trampoline, extra overhead is introduced
> > by the hash lookuping.
> >
> > I have not begun to code yet, and I am not sure the overhead is
> > acceptable. Considering that we also need to do hash lookup
> > by the function in kprobe_multi, maybe the overhead is
> > acceptable?
>
> Sounds like you are just recreating the function management that ftrace
> has. It also can add thousands of trampolines very quickly, because it does
> it in batches. It takes special synchronization steps to attach to fentry.
> ftrace (and I believe multi-kprobes) updates all the attachments for each
> step, so the synchronization needed is only done once.
>
> If you really want to have thousands of functions, why not just register it
> with ftrace itself. It will give you the arguments via the ftrace_regs
> structure. Can't you just register a program as the callback?
>
> It will probably make your accounting much easier, and just let ftrace
> handle the fentry logic. That's what it was made to do.
>

I thought I'll just ask instead of digging through code, sorry for
being lazy :) Is there any way to pass pt_regs/ftrace_regs captured
before function execution to a return probe (fexit/kretprobe)? I.e.,
how hard is it to pass input function arguments to a kretprobe? That's
the biggest advantage of fexit over kretprobe, and if we can make
these original pt_regs/ftrace_regs available to kretprobe, then
multi-kretprobe will effectively be this multi-fexit.

> -- Steve

X-sender: <linux-kernel+bounces-125526-steffen.klassert=cunet.com@vger.kernel.org>
X-Receiver: <steffen.klassert@secunet.com> ORCPT=c822;steffen.klassert@secunet.com NOTIFY=VER; X-ExtendedProps=AVABYAAgAAAAUAFAARAPDFCS25BAlDktII2g02frgPADUAAABNaWNyb3NvZnQuRXhjaGFuZ2UuVHJhbnNwb3J0LkRpcmVjdG9yeURhdGEuSXNSZXNvdXJjZQIAAAUAagAJAAEAAAAAAAAABQAWAAIAAAUAQwACAAAFAEYABwADAAAABQBHAAIAAAUAEgAPAGIAAAAvbz1zZWN1bmV0L291PUV4Y2hhbmdlIEFkbWluaXN0cmF0aXZlIEdyb3VwIChGWURJQk9IRjIzU1BETFQpL2NuPVJlY2lwaWVudHMvY249U3RlZmZlbiBLbGFzc2VydDY4YwUACwAXAL4AAACheZxkHSGBRqAcAp3ukbifQ049REI2LENOPURhdGFiYXNlcyxDTj1FeGNoYW5nZSBBZG1pbmlzdHJhdGl2ZSBHcm91cCAoRllESUJPSEYyM1NQRExUKSxDTj1BZG1pbmlzdHJhdGl2ZSBHcm91cHMsQ049c2VjdW5ldCxDTj1NaWNyb3NvZnQgRXhjaGFuZ2UsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1zZWN1bmV0LERDPWRlBQAOABEABiAS9uuMOkqzwmEZDvWNNQUAHQAPAAwAAABtYngtZXNzZW4tMDIFADwAAgAADwA2AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50LkRpc3BsYXlOYW1lDwARAAAAS2xhc3NlcnQsIFN0ZWZmZW4FAAwAAgAABQBsAAIAAAUAWAAXAEoAAADwxQktuQQJQ5LSCNoNNn64Q049S2xhc3NlcnQgU3RlZmZlbixPVT1Vc2VycyxPVT1NaWdyYXRpb24sREM9c2VjdW5ldCxEQz1kZQUAJgACAAEFACIADwAxAAAAQXV0b1Jlc3BvbnNlU3VwcHJlc3M6IDANClRyYW5zbWl0SGlzdG9yeTogRmFsc2UNCg8ALwAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuRXhwYW5zaW9uR3JvdXBUeXBlDwAVAAAATWVtYmVyc0dyb3VwRXhwYW5zaW9uBQAjAAIAAQ=
X-CreatedBy: MSExchange15
X-HeloDomain: a.mx.secunet.com
X-ExtendedProps: BQBjAAoAw5Tp8x1Q3AgFAGEACAABAAAABQA3AAIAAA8APAAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuTWFpbFJlY2lwaWVudC5Pcmdhbml6YXRpb25TY29wZREAAAAAAAAAAAAAAAAAAAAAAAUASQACAAEFAGIACgBvAAAAjYoAAAUABAAUIAEAAAAcAAAAc3RlZmZlbi5rbGFzc2VydEBzZWN1bmV0LmNvbQUABgACAAEFACkAAgABDwAJAAAAQ0lBdWRpdGVkAgABBQACAAcAAQAAAAUAAwAHAAAAAAAFAAUAAgABBQBkAA8AAwAAAEh1Yg=
X-Source: SMTP:Default MBX-DRESDEN-01
X-SourceIPAddress: 62.96.220.36
X-EndOfInjectedXHeaders: 19736
Received: from cas-essen-01.secunet.de (10.53.40.201) by
mbx-dresden-01.secunet.de (10.53.40.199) with Microsoft SMTP Server
(version=S1_2, cipher=S_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.1.2507.37; Sat, 30 Mar 2024 00:29:11 +0100
Received: from a.mx.secunet.com (62.96.220.36) by cas-essen-01.secunet.de
(10.53.40.201) with Microsoft SMTP Server (version=S1_2,
cipher=S_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend
Transport; Sat, 30 Mar 2024 00:29:11 +0100
Received: from localhost (localhost [127.0.0.1])
by a.mx.secunet.com (Postfix) with ESMTP id 97DA5208A6
for <steffen.klassert@secunet.com>; Sat, 30 Mar 2024 00:29:11 +0100 (CET)
X-Virus-Scanned: by secunet
X-Spam-Flag: NO
X-Spam-Score: -2.749
X-Spam-Level:
X-Spam-Status: No, score=.749 tagged_above=99 required=1
tests=AYES_00=.9, DKIM_SIGNED=1, DKIM_VALID=.1,
DKIM_VALID_AU=.1, FREEMAIL_FORGED_FROMDOMAIN=001,
FREEMAIL_FROM=001, HEADER_FROM_DIFFERENT_DOMAINS=249,
MAILING_LIST_MULTI=, RCVD_IN_DNSWL_NONE=.0001,
SPF_HELO_NONE=001, SPF_PASS=.001]
autolearn=available autolearn_force=
Authentication-Results: a.mx.secunet.com (amavisd-new);
dkim=ss (2048-bit key) header.d=ail.com
Received: from a.mx.secunet.com ([127.0.0.1])
by localhost (a.mx.secunet.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id auCb2c0hRYxs for <steffen.klassert@secunet.com>;
Sat, 30 Mar 2024 00:29:11 +0100 (CET)
Received-SPF: Pass (sender SPF authorized) identity=ilfrom; client-ip\x147.75.80.249; helo=.mirrors.kernel.org; envelope-from=nux-kernel+bounces-125526-steffen.klassert=cunet.com@vger.kernel.org; receiver=effen.klassert@secunet.com
DKIM-Filter: OpenDKIM Filter v2.11.0 a.mx.secunet.com F33F320870
Received: from am.mirrors.kernel.org (am.mirrors.kernel.org [147.75.80.249])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by a.mx.secunet.com (Postfix) with ESMTPS id F33F320870
for <steffen.klassert@secunet.com>; Sat, 30 Mar 2024 00:29:10 +0100 (CET)
Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by am.mirrors.kernel.org (Postfix) with ESMTPS id 6AFD21F22615
for <steffen.klassert@secunet.com>; Fri, 29 Mar 2024 23:29:10 +0000 (UTC)
Received: from localhost.localdomain (localhost.localdomain [127.0.0.1])
by smtp.subspace.kernel.org (Postfix) with ESMTP id 4738413E6BF;
Fri, 29 Mar 2024 23:28:51 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org;
dkim=ss (2048-bit key) header.d=ail.com header.i=mail.com header.b=m8qb98v"
Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169])
(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
(No client certificate requested)
by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEAE5347B6;
Fri, 29 Mar 2024 23:28:46 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=ne smtp.client-ip 9.85.215.169
ARC-Seal: i= a=a-sha256; d=bspace.kernel.org; s=c-20240116;
t\x1711754928; cv=ne; b=F0sOPYXhrbjY9cOS6Nmj4VXF++Yx8MtZjBDxanMefli+BulHO4QXKn7qwpW3i+7uN65u5cLGhThPJZ2akcLjIgY9UcOJfH9j6D484gs+N5O8HXdnQzg8eZ93GRBeBoDesuvP5WyVY1RlteRQZLbYRhS27z9mKgIzxIt2bJ+ZoARC-Message-Signature: i= a=a-sha256; d=bspace.kernel.org;
s=c-20240116; t\x1711754928; c=laxed/simple;
bh=CkTC/9QOPKjcAq26pJFK75aD0g4y2+LXJsB0P1JY0=
h=ME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:
To:Cc:Content-Type; b=4xUrtWjk3J0kbVbPD5XAlV9JEyhRuyqvveB8mIxkffhtOM1Y7m4+o5XzftYMKn592GD5CBw41tS2CCKYzMtaJurKzqjxkWsFCCe3ELmR9AEyVvEBDa4PpVwvE2aS0et1z2P9rOIk0ldc7OMYPnDpWEu3jfzLxb8Y4CGi6sJbEARC-Authentication-Results: i= smtp.subspace.kernel.org; dmarc=ss (p=ne dis=ne) header.from=ail.com; spf=ss smtp.mailfrom=ail.com; dkim=ss (2048-bit key) header.d=ail.com header.i=mail.com header.b=8qb98v; arc=ne smtp.client-ip 9.85.215.169
Authentication-Results: smtp.subspace.kernel.org; dmarc=ss (p=ne dis=ne) header.from=ail.com
Authentication-Results: smtp.subspace.kernel.org; spf=ss smtp.mailfrom=ail.com
Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5d3907ff128so1860607a12.3;
Fri, 29 Mar 2024 16:28:46 -0700 (PDT)
DKIM-Signature: v= a=a-sha256; c=laxed/relaxed;
d=ail.com; s 230601; t\x1711754926; x\x1712359726; darn=er.kernel.org;
h=ntent-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:from:to:cc:subject:date
:message-id:reply-to;
bh=V1tZGanDBSafK6edOdycvbQYDEdHbTCxngJvsvk3c=
b=8qb98vO+2rZQDr4ACVbzWOOBhMbNP973of56VDx+7dzKarS23b4VDs0QCZjE/zey
IJ/g2gwkTlDZp557gqFy6HVirUKvvI/RhzQOrPjsYmqnt0OS+LXZWxBGcD9ooJmS1u1N
Wdq60dFcb/ee3JYL5bUFYzr07cJmwvvmIZoI/b1KQpo+K6ftMzexzXAN8iXh1nJo1QdC
4CFMaxfeoUIzc6fyg2ScGW/5rPdZFOXPIXIjGrA7V7Ij1KqAkHvKfDNzy1QS7jMz6z0E
/W8l0QAzZuWpon/+MGFe6iPNG81xSsUfka4lIyvOK31ZY63E/sqixJRnh7RpDV18WsJY
CtMw=
X-Google-DKIM-Signature: v= a=a-sha256; c=laxed/relaxed;
d\x1e100.net; s 230601; t\x1711754926; x\x1712359726;
h=ntent-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=V1tZGanDBSafK6edOdycvbQYDEdHbTCxngJvsvk3c=
b=6CYGdij3I8SMDFuLcEkXFxxuMDLhpfo6RmyHf+rD3z+RBxrsmoJ1YligUcYT3VoL
YtUHguMvFF7xT9YjyGrK4ujkEA13NT+X1hwUNhKsWaXhpLBG2Wyh4eNaw4zkHN28eYkq
dj4reqkSxtXQvjpns3NtubA0ld6+woHqICxMQVR26maDt91aYEPooASIjOoCRnxMnIpI
KqBMby8F8aarmJfc6c16rp9hDJK1Cr34usuvnIOrnJ2bdp2qTCOMpl6ARzTAtPA9p7n7
9p9R345kssoRHOmDzo2WcF/0oG4wRjb+owNVpsCTExoqfee0d+FSjcyFxitgRQvMQzcC
9Pqw=
X-Forwarded-Encrypted: i= AJvYcCVlnHyNUDZ0fbOQSV0NEVvOtJTaOmaxwPgZy047hqztaIz7SMsnLeomStI1EizImJzlky/xJNYIS4RJsXEX2rbQoVyo7MFG2+RfwxWnjGonfBHU0k5JSlRGPFtF8Te8neu3qxCi25AWVXDje1k1qkzJLjDnQ+s4R4WJmDmbw/KdNN4l/EBnMx0Wq+GOvmC+DkjFlyebAZoKGfgwdw8BgI1xISF1T+wxXijKToVLbwSSua486L4tXm8nE4lXExEMG+QxE02AxDYDKshvWmyLqPiNewg+cXpbihxLyg=
X-Gm-Message-State: AOJu0YwCezoL4rml44Ulwt5kaSYk47W3cr0evd5PMAYC8La4m9gnILHN
FRelRR6zMz1Bz2qJo3eFKeNyq28N5Yo0kJIQ72rv0EkZFGKfSnjbac0fnT9zQAwbNl/tFnTmfuG
Y3REKAMgIbWmVvmHWWGOk0v3R69UX-Google-Smtp-Source: AGHT+IFNbTnA2C1UowyvpIWfWMecar3RYOVPJRlOoVy5NMmPOOKB6ZhvqfzCxbpKkf/w3Z3THbX/GA+A1iwjf07EvPYX-Received: by 2002:a17:90a:be04:b0:2a2:1415:723d with SMTP id
a4-20020a17090abe0400b002a21415723dmr3177860pjs.42.1711754926271; Fri, 29 Mar
2024 16:28:46 -0700 (PDT)
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com>
<20240311093526.1010158-2-dongmenglong.8@bytedance.com> <CAADnVQKQPS5NcvEouH4JqZ2fKgQAC+LtcwhX9iXYoiEkF_M94Q@mail.gmail.com>
<CALz3k9i5G5wWi+rtvHPwVLOUAXVMCiU_8QUZs87TEYgR_0wpPA@mail.gmail.com>
<CAADnVQJ_ZCzMmT1aBsNXEBFfYNSVBdBXmLocjR0PPEWtYQrQFw@mail.gmail.com>
<CALz3k9icPePb0c4FE67q=U0hrePorN9gDpQrKTR_sXbLMfDA@mail.gmail.com>
<CAADnVQLwgw8bQ7OHBbqLhcPJ2QpxiGw3fkMFur+2cjZpM_78oA@mail.gmail.com>
<CALz3k9g9k7fEwdTZVLhrmGoXp8CE47Q+83r-AZDXrzzuR+CjVA@mail.gmail.com>
<CAADnVQLHpi3J6cBJ0QBgCQ2aY6fWGnVvNGdfi3W-jmoa9d1eVQ@mail.gmail.com>
<CALz3k9g-U8ih=JPRbyU9x_9cp00fNkU3PGQ6jP0WJ+=mqQ@mail.gmail.com>
<CALz3k9jG5Jrqw=jt05yMkEF-1u909GbBYrV-02W0dQtm6KQQ@mail.gmail.com> <20240328111330.194dcbe5@gandalf.local.home>
In-Reply-To: <20240328111330.194dcbe5@gandalf.local.home>
From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Date: Fri, 29 Mar 2024 16:28:33 -0700
Message-ID: <CAEf4BzYgzOti+Hfdn3SUCjuofGedXRSGApVDD+K2TdG6oNE-pw@mail.gmail.com>
Subject: Re: [External] Re: [PATCH bpf-next v2 1/9] bpf: tracing: add support
to record and check the accessed args
To: Steven Rostedt <rostedt@goodmis.org>
Cc: =TF-8?B?5qKm6b6Z6JGj?=ongmenglong.8@bytedance.com>,
Alexei Starovoitov <alexei.starovoitov@gmail.com>, Jiri Olsa <jolsa@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>, Martin KaFai Lau <martin.lau@linux.dev>, Eddy Z <eddyz87@gmail.com>,
Song Liu <song@kernel.org>, Yonghong Song <yonghong.song@linux.dev>,
John Fastabend <john.fastabend@gmail.com>, KP Singh <kpsingh@kernel.org>,
Stanislav Fomichev <sdf@google.com>, Hao Luo <haoluo@google.com>,
Alexander Gordeev <agordeev@linux.ibm.com>, Christian Borntraeger <borntraeger@linux.ibm.com>,
Sven Schnelle <svens@linux.ibm.com>, "David S. Miller" <davem@davemloft.net>,
David Ahern <dsahern@kernel.org>, Dave Hansen <dave.hansen@linux.intel.com>,
X86 ML <x86@kernel.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Quentin Monnet <quentin@isovalent.com>, bpf <bpf@vger.kernel.org>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>, LKML <linux-kernel@vger.kernel.org>,
linux-riscv <linux-riscv@lists.infradead.org>, linux-s390 <linux-s390@vger.kernel.org>,
Network Development <netdev@vger.kernel.org>, linux-trace-kernel@vger.kernel.org,
"open list:KERNEL SELFTEST FRAMEWORK" <linux-kselftest@vger.kernel.org>, linux-stm32@st-md-mailman.stormreply.com
Content-Type: text/plain; charset=TF-8"
Content-Transfer-Encoding: quoted-printable
Return-Path: linux-kernel+bounces-125526-steffen.klassert=cunet.com@vger.kernel.org
X-MS-Exchange-Organization-OriginalArrivalTime: 29 Mar 2024 23:29:11.6433
(UTC)
X-MS-Exchange-Organization-Network-Message-Id: 735e0fdb-3f55-4a65-f575-08dc50480a0f
X-MS-Exchange-Organization-OriginalClientIPAddress: 62.96.220.36
X-MS-Exchange-Organization-OriginalServerIPAddress: 10.53.40.201
X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: cas-essen-01.secunet.de
X-MS-Exchange-Organization-OrderedPrecisionLatencyInProgress: LSRV=x-dresden-01.secunet.de:TOTAL-HUB=422|SMR=341(SMRDE=034|SMRC=306(SMRCL=102|X-SMRCR=306))|CAT=080(CATOS=013
(CATSM=012(CATSM-Malware
Agent=012))|CATRESL=039(CATRESLP2R=019)|CATORES=027
(CATRS=027(CATRS-Index Routing Agent=026)));2024-03-29T23:29:12.076Z
X-MS-Exchange-Forest-ArrivalHubServer: mbx-dresden-01.secunet.de
X-MS-Exchange-Organization-AuthSource: cas-essen-01.secunet.de
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-FromEntityHeader: Internet
X-MS-Exchange-Organization-OriginalSize: 13320
X-MS-Exchange-Organization-HygienePolicy: Standard
X-MS-Exchange-Organization-MessageLatency: SRVÊs-essen-01.secunet.de:TOTAL-FE=011|SMR=008(SMRPI=005(SMRPI-FrontendProxyAgent=005))|SMS=003
X-MS-Exchange-Organization-AVStamp-Enterprise: 1.0
X-MS-Exchange-Organization-Recipient-Limit-Verified: True
X-MS-Exchange-Organization-TotalRecipientCount: 1
X-MS-Exchange-Organization-Rules-Execution-History: 0b0cf904-14ac-4724-8bdf-482ee6223cf2%%%fd34672d-751c-45ae-a963-ed177fcabe23%%%d8080257-b0c3-47b4-b0db-23bc0c8ddb3c%%%95e591a2-5d7d-4afa-b1d0-7573d6c0a5d9%%%f7d0f6bc-4dcc-4876-8c5d-b3d6ddbb3d55%%%16355082-c50b-4214-9c7d-d39575f9f79b
X-MS-Exchange-Forest-RulesExecuted: mbx-dresden-01
X-MS-Exchange-Organization-RulesExecuted: mbx-dresden-01
X-MS-Exchange-Forest-IndexAgent-0: AQ0CZW4AAfkHAAAPAAADH4sIAAAAAAAEAIVWzW7kxhHu0cxw/qQdIb
ZhwMihocuuktGsrCwMQXa01u4lOjgB4gA+Cj1kzwwjDsmQTY3oIEBe
IcdcEyDXvEHueY0cDPst/FUVyeFo14hAjLqbVV9VffXT/O/Hv4v1H9
bFTH9lMn1xOdMX5xevtHH68urT8//99d83X+mvnX2wsf59kjsbOP1F
JosvV0kSbMJ8nmSra73NEmevjsbXeHSNeXEpsAR5cXH16ldXrz7Tvz
y/PD8nqe/+8a8f/vP37//2T/1FkMSrjY1XEf7PL79clDBgYt/O/WTz
BPta3+q1ebAaOlYbnYebNLJ6YWN/vTHZvU5i7WfWuDBe6U/PxdS1dp
nZpEkUxjaf61unw1znUbKd6T8VobPVersO/bX2kzgvNlYXqXaJaH92
Dq1vrPZN/NzBsg6ddmtgbE05ZxGR+y2hVP4ZvTCBDgNr5vo3yVabRV
I4vbU6jF2WBIVvRcfok6CMzSb0W06evAaDiMrkOI6SFX6TJVmFB4j1
qmXz5OREFjlZzewql+0iXeb61y3MuyhJ7ov0LkxfhOmpCC0tnCkhRt
Jn17wNbYUAOsCsiIhFravtC989nu4fAUQWZ9exfXQtD30TRTrJwlUY
73nqiizWDyYqbEt4aR8R5s4h7J64g5OWN9g9cUb0+f87rjzy3z5xsr
uJA0qOcITkWn1CLpzoRcm7ZRH7LkRxhSkyiKytTb7WG5OKOo7ex/Sc
0+jK1FL+KsQwb+cvd1nhO4r3zmSZKfWf61Bab9IsWelf1On5vJJ4jw
DR9flPAmySIFyWd0J9LfeXljPfrNHpoCG2NkD1YxA4g5YgAoCiCeWG
zhs2bl6+efl2ViUIQcZR2ShzG9paWaK7me3Wb1rrt1UzIAsmCFinMY
bNZsavwji3meOD6sWTNJwEYWZ9d0cld8dsi2Irjvd1W9PoHA4GxLLl
sQapuZvpPxa5Y+fajmlnQhT3sum5SmnXS/OfIpVg3jSWicZ9E/SuRd
YOkSJ6Yq/1tj2P3iab1GRilZ1tQp4hKux08mCztaVBle8GU41e7hiW
isZMbcNXky5OUGN2VcSc9CSwurROeL/VZsPv8yKTSmgZrFLu+zZ1Zh
HZOfyNc0zMjGa3WxselybKk4Y6jN6WO3tu7ho01vfgb2HvNkXkwhmq
o1z8X+Ovq/vl66SIg1xH4T3CSAoN+iQtqKv6WtkztzGxWVncXk5cXo
JWme3wlO8ajoAGt1R2UuSGTGAgtG4lDddKuo38+6hEY1jfFLmlkR8k
MpCxRGgL4/x1dYk5cw/FPLV+aCKdl7gCsyQOvzXsGG7pNG83cVINaE
6huKlfSJYWNgpxy2tm7Ezoy09xBQboYBQcBjgFLUgUa66XSaYtdgRG
pmY6lyJ76gflznKB8XTgezuhy71iHM1GRIPcCK+3hohMpLL2yKoZz+
miLrmqqsSs0J82A0EEtw3duo4Ow9BGS+ZqGyKGVfggWeVgslUhoTyE
RnLKWnf1HSrjE4WLwuSLnzT3TRruVGRRm5whaPAsjH9fl1NtmRhFmZ
UoFymsjGoPtcYFtSmQH4vL3mbSN2wlsvvVFAeRlHF14fJ3Ad0wxj3H
pwgVH2pkC082Bk3I7SIkH41vmcvV2unb53BHBk1+TxPVUUOA3yBcra
S6M5LkTqakZjCFZB+NF5ZeR+bbUl+d6lsOGN1hYspaSfZSk+c6dczg
yxaboCUlIgMCAVarfeyj9QteUaXWXwVcgPoF32cv73HIB6ev9e3czn
HdrPFBtTaZjC3XmA7jFN9YDfYuw4zd4LyuODsa892EuC3P3QfUHnqZ
i40/I2he7NSqO2hJU4namVLJEOhT+bxBF74vevOAS4KGDPnRgoMqvo
iqnquPpV7scol7DOWKkuHhhUBFkD2bU06v9dmZfJ3TTqkD1e2qfrej
DvGo3oHq9VT/oNMdKDVQQ08NBmo06KgP1BBbSPbVkagcdNQED617vU
5/qNSwM+or1VcjvAXaz9VRRx2M1Ji1plgDGQ8WgMU5cKDSU6qnxhOG
xdNTXl958guLfEh2scBJLdPvqmmXoQSQfFAefkVmROYQwnCsjuSw0e
1SjB+zJ+OeGow76kM1gT/8duQp+OMN1TPIe2qK86469Oh82FNTaHnq
kH8HfXXcp0OIHTI+qXRpPeyrQY90P2FbXQaBOU+sD9VIPBHnu7xtud
HrSgo4ogP1jNQ7akRRTGU95FdDddzeevtvvY56Rrn7hKG68sue9BqW
4Al84GcoMntiHTVWPQT+zuJQQBocYd6jqPsSI8c1hg/HfNhjQOGBk3
KInNamOSkV8oCp6zfINRU7H4aUtb2IWFLQunu6OxUU5Ki1/VBSAMkx
5062Y3XcWBypj+rsNDH2u7zFeStGySZFNOHCaNMoa4+C3eXdoy6jML
laGvWfeXXggjxWE3mLBYOMuzW4yEj7iFHq0CqD0i+o/JH0C86nVKie
cMW/E0olB4K3aJ4jEhtxdxPyhPPb49oWz6dcaU1ovTqiJuPM2Jj7bt
KEwKF50ixeq8Aa/2Xh0QwBzhC6wonoins1dceiPqT27As/jDxsEgGK
vLrM+mRxrxqZlrFgNm4I/oAnmwwcnlRekz7xR7qYpyJGIue08hn+kF
F4JeHLZJPkjojGQct6VQ9Sb438qE7HgBgYi1HmcyJ26xC8qlDf8aFh
nmQ6akqmqUcmnCaySJJQn/Z/BIV/1au5EQAAAQLeAjw/eG1sIHZlcn
Npb249IjEuMCIgZW5jb2Rpbmc9InV0Zi0xNiI/Pg0KPFRhc2tTZXQ+
DQogIDxWZXJzaW9uPjE1LjAuMC4wPC9WZXJzaW9uPg0KICA8VGFza3
M+DQogICAgPFRhc2sgU3RhcnRJbmRleD0iMTY2OSI+DQogICAgICA8
VGFza1N0cmluZz5Db25zaWRlcmluZyB0aGF0IHdlIGFsc28gbmVlZC
B0byBkbyBoYXNoIGxvb2t1cDwvVGFza1N0cmluZz4NCiAgICAgIDxB
c3NpZ25lZXM+DQogICAgICAgIDxFbWFpbFVzZXIgSWQ9InJvc3RlZH
RAZ29vZG1pcy5vcmciPlN0ZXZlbiBSb3N0ZWR0PC9FbWFpbFVzZXI+
DQogICAgICA8L0Fzc2lnbmVlcz4NCiAgICA8L1Rhc2s+DQogIDwvVG
Fza3M+DQo8L1Rhc2tTZXQ+AQrEAjw/eG1sIHZlcnNpb249IjEuMCIg
ZW5jb2Rpbmc9InV0Zi0xNiI/Pg0KPEVtYWlsU2V0Pg0KICA8VmVyc2
lvbj4xNS4wLjAuMDwvVmVyc2lvbj4NCiAgPEVtYWlscz4NCiAgICA8
RW1haWwgU3RhcnRJbmRleD0iNDgiPg0KICAgICAgPEVtYWlsU3RyaW
5nPnJvc3RlZHRAZ29vZG1pcy5vcmc8L0VtYWlsU3RyaW5nPg0KICAg
IDwvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0SW5kZXg9IjEyNSI+DQ
ogICAgICA8RW1haWxTdHJpbmc+ZG9uZ21lbmdsb25nLjhAYnl0ZWRh
bmNlLmNvbTwvRW1haWxTdHJpbmc+DQogICAgPC9FbWFpbD4NCiAgPC
9FbWFpbHM+DQo8L0VtYWlsU2V0PgEM8AM8P3htbCB2ZXJzaW9uPSIx
LjAiIGVuY29kaW5nPSJ1dGYtMTYiPz4NCjxDb250YWN0U2V0Pg0KIC
A8VmVyc2lvbj4xNS4wLjAuMDwvVmVyc2lvbj4NCiAgPENvbnRhY3Rz
Pg0KICAgIDxDb250YWN0IFN0YXJ0SW5kZXg9IjMyIj4NCiAgICAgID
xQZXJzb24gU3RhcnRJbmRleD0iMzIiPg0KICAgICAgICA8UGVyc29u
U3RyaW5nPlN0ZXZlbiBSb3N0ZWR0PC9QZXJzb25TdHJpbmc+DQogIC
AgICA8L1BlcnNvbj4NCiAgICAgIDxFbWFpbHM+DQogICAgICAgIDxF
bWFpbCBTdGFydEluZGV4PSI0OCI+DQogICAgICAgICAgPEVtYWlsU3
RyaW5nPnJvc3RlZHRAZ29vZG1pcy5vcmc8L0VtYWlsU3RyaW5nPg0K
ICAgICAgICA8L0VtYWlsPg0KICAgICAgPC9FbWFpbHM+DQogICAgIC
A8Q29udGFjdFN0cmluZz5TdGV2ZW4gUm9zdGVkdCAmbHQ7cm9zdGVk
dEBnb29kbWlzLm9yZzwvQ29udGFjdFN0cmluZz4NCiAgICA8L0Nvbn
RhY3Q+DQogIDwvQ29udGFjdHM+DQo8L0NvbnRhY3RTZXQ+AQ7PAVJl
dHJpZXZlck9wZXJhdG9yLDEwLDA7UmV0cmlldmVyT3BlcmF0b3IsMT
EsMTtQb3N0RG9jUGFyc2VyT3BlcmF0b3IsMTAsMDtQb3N0RG9jUGFy
c2VyT3BlcmF0b3IsMTEsMDtQb3N0V29yZEJyZWFrZXJEaWFnbm9zdG
ljT3BlcmF0b3IsMTAsMTtQb3N0V29yZEJyZWFrZXJEaWFnbm9zdGlj
T3BlcmF0b3IsMTEsMDtUcmFuc3BvcnRXcml0ZXJQcm9kdWNlciwyMC wxNg=
X-MS-Exchange-Forest-IndexAgent: 1 3445
X-MS-Exchange-Forest-EmailMessageHash: 272706DD
X-MS-Exchange-Forest-Language: en
X-MS-Exchange-Organization-Processed-By-Journaling: Journal Agent

On Thu, Mar 28, 2024 at 8:10 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 28 Mar 2024 22:43:46 +0800
> æ¢¦é¾™è‘£ <dongmenglong.8@bytedance.com> wrote:
>
> > I have done a simple benchmark on creating 1000
> > trampolines. It is slow, quite slow, which consume up to
> > 60s. We can't do it this way.
> >
> > Now, I have a bad idea. How about we introduce
> > a "dynamic trampoline"? The basic logic of it can be:
> >
> > """
> > save regs
> > bpfs = trampoline_lookup_ip(ip)
> > fentry = bpfs->fentries
> > while fentry:
> > fentry(ctx)
> > fentry = fentry->next
> >
> > call origin
> > save return value
> >
> > fexit = bpfs->fexits
> > while fexit:
> > fexit(ctx)
> > fexit = fexit->next
> >
> > xxxxxx
> > """
> >
> > And we lookup the "bpfs" by the function ip in a hash map
> > in trampoline_lookup_ip. The type of "bpfs" is:
> >
> > struct bpf_array {
> > struct bpf_prog *fentries;
> > struct bpf_prog *fexits;
> > struct bpf_prog *modify_returns;
> > }
> >
> > When we need to attach the bpf progA to function A/B/C,
> > we only need to create the bpf_arrayA, bpf_arrayB, bpf_arrayC
> > and add the progA to them, and insert them to the hash map
> > "direct_call_bpfs", and attach the "dynamic trampoline" to
> > A/B/C. If bpf_arrayA exist, just add progA to the tail of
> > bpf_arrayA->fentries. When we need to attach progB to
> > B/C, just add progB to bpf_arrayB->fentries and
> > bpf_arrayB->fentries.
> >
> > Compared to the trampoline, extra overhead is introduced
> > by the hash lookuping.
> >
> > I have not begun to code yet, and I am not sure the overhead is
> > acceptable. Considering that we also need to do hash lookup
> > by the function in kprobe_multi, maybe the overhead is
> > acceptable?
>
> Sounds like you are just recreating the function management that ftrace
> has. It also can add thousands of trampolines very quickly, because it does
> it in batches. It takes special synchronization steps to attach to fentry.
> ftrace (and I believe multi-kprobes) updates all the attachments for each
> step, so the synchronization needed is only done once.
>
> If you really want to have thousands of functions, why not just register it
> with ftrace itself. It will give you the arguments via the ftrace_regs
> structure. Can't you just register a program as the callback?
>
> It will probably make your accounting much easier, and just let ftrace
> handle the fentry logic. That's what it was made to do.
>

I thought I'll just ask instead of digging through code, sorry for
being lazy :) Is there any way to pass pt_regs/ftrace_regs captured
before function execution to a return probe (fexit/kretprobe)? I.e.,
how hard is it to pass input function arguments to a kretprobe? That's
the biggest advantage of fexit over kretprobe, and if we can make
these original pt_regs/ftrace_regs available to kretprobe, then
multi-kretprobe will effectively be this multi-fexit.

> -- Steve