From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 129611FC4; Fri, 26 Apr 2024 16:55:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714150505; cv=none; b=rIipwmanEbBpIs5dGT27L8cXre58J2BpTm9iHFkRrZb9zM8jDBnswRZh2yBroOvCdypu+DX39l1SAws+NOVd2Q3UPJds7wy4KicrcjT7uToBUl7jTHmOC1ES4zzT3G6bo/QhICsmFxLECLNUfuobewyhCYqPKpgGNlTk26gmp3U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714150505; c=relaxed/simple; bh=FgJb8ip6P9+AklWn3gKziPHPVzIvMpCQOIrfmY6RwzA=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=XT2kfPz+n9jjgoM9WJmqYhXIxhCZ0EcDILcXXBYYu35abhhD8oZ7XGnK3vP59JIITVmNv5Cpixk3mYzgvhNDP7H7TR5PfOcwktu1AdrsBqv71GvcPRtrTp/dTZ2Ce9PNtOs0ZdUbf0kFnQwDhQsaP3fbNQ/pqThSlRduRW9/sgc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vA4G+RUS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vA4G+RUS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 50C9CC113CD; Fri, 26 Apr 2024 16:55:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714150504; bh=FgJb8ip6P9+AklWn3gKziPHPVzIvMpCQOIrfmY6RwzA=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=vA4G+RUS2HCJjpvDO8FpBD1l0DUJmIj8idLllzwfJsLdQt5/XX8873dkUW6c5xMI5 quEPkGlllVKbBheAhqwwkDJozCkwTrWYn6p2MO/WnKhnib7JQKC9ayvlAgw3EguLvy GuXmVKwv3RvIsafjIxleQBNT9As4DNDGlh4+3bb94zTQlCDTJuCAUknG05Nkqbrh9S DGkgEAQgTf5NazN86rOfIgfJBwtIaDibaEYlB7BCdxYzE0w+RL5h6X6zG7hC5oDA6c eUeW7IXR9DgBVL3m5ZhW/oaNbdbcYLObeIIrX6L6gDbTqOxPGOepvNHWII6c/hJgZO I1zf8zCgscUdQ== From: Puranjay Mohan To: Andrii Nakryiko Cc: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: Re: [PATCH bpf-next v3 1/2] arm64, bpf: add internal-only MOV instruction to resolve per-CPU addrs In-Reply-To: References: <20240426121349.97651-1-puranjay@kernel.org> <20240426121349.97651-2-puranjay@kernel.org> Date: Fri, 26 Apr 2024 16:55:01 +0000 Message-ID: Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Andrii Nakryiko writes: > On Fri, Apr 26, 2024 at 5:14=E2=80=AFAM Puranjay Mohan wrote: >> >> From: Puranjay Mohan >> >> Support an instruction for resolving absolute addresses of per-CPU >> data from their per-CPU offsets. This instruction is internal-only and >> users are not allowed to use them directly. They will only be used for >> internal inlining optimizations for now between BPF verifier and BPF >> JITs. >> >> Since commit 7158627686f0 ("arm64: percpu: implement optimised pcpu >> access using tpidr_el1"), the per-cpu offset for the CPU is stored in >> the tpidr_el1/2 register of that CPU. >> >> To support this BPF instruction in the ARM64 JIT, the following ARM64 >> instructions are emitted: >> >> mov dst, src // Move src to dst, if src !=3D dst >> mrs tmp, tpidr_el1/2 // Move per-cpu offset of the current cpu in tmp. >> add dst, dst, tmp // Add the per cpu offset to the dst. >> >> To measure the performance improvement provided by this change, the >> benchmark in [1] was used: >> >> Before: >> glob-arr-inc : 23.597 =C2=B1 0.012M/s >> arr-inc : 23.173 =C2=B1 0.019M/s >> hash-inc : 12.186 =C2=B1 0.028M/s >> >> After: >> glob-arr-inc : 23.819 =C2=B1 0.034M/s >> arr-inc : 23.285 =C2=B1 0.017M/s > > I still expected a better improvement (global-arr-inc's results > improved more than arr-inc, which is completely different from > x86-64), but it's still a good thing to support this for arm64, of > course. > > ack for generic parts I can understand: > > Acked-by: Andrii Nakryiko > I will have to do more research to find why we don't see very high improvement. But this is what is happening here: This was the complete picture before inlining: int cpu =3D bpf_get_smp_processor_id(); mov x10, #0xffffffffffffd4a8 movk x10, #0x802c, lsl #16 movk x10, #0x8000, lsl #32 blr x10 ---------------------------------------> nop nop adrp x0, 0xffff8000= 82128000 mrs x1, tpidr_el1 add x0, x0, #0x8 ldrsw x0, [x0, x1] <----------------------------------------ret add x7, x0, #0x0 Now we have: int cpu =3D bpf_get_smp_processor_id(); mov x7, #0xffff8000ffffffff movk x7, #0x8212, lsl #16 movk x7, #0x8008 mrs x10, tpidr_el1 add x7, x7, x10 ldr w7, [x7] So, we have removed multiple instructions including a branch and a return. I was expecting to see more improvement. This benchmark is taken from a KVM based virtual machine, maybe if I do it on bare-metal I would see more improvement ? Thanks, Puranjay From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E194C4345F for ; Fri, 26 Apr 2024 16:55:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qlxhL7xDZS5vnYTUZqGqMaN9MDY+llDzJ00DXLLMa3w=; b=bCLZ2nz7K7dxZs Nz4AKnZU9m1KLq41zGiP138JgSy4jsy3VQovNwDzoQ3Ntn4/vsSA6XvZSFRjEbP6+Mtqlpy59AZFO WuK4NrtjVIV8uwngMoj5n7Y30bb/pBSrXd80uSuxH6ESBwIvqKqz41pjP9ljZwdDoZVCk9mvBnAPu Q9384dv4aXr7J5REqC6DdlPNoGvWPdXuLsFH7khtxarIPwyZBOU6wqkUz9/3F7G8Y9zQk6b2mYkgM zPO3U75JDKkeytXJL1C5zF0IRLem+G+cFKL1p2oZdbsXuOGjVWD3KbcHbwW0fH+y8G2xpvAsd8DBH MqRqFXQ/JFdlS3bPytwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0Oqt-0000000DLRz-2tMG; Fri, 26 Apr 2024 16:55:11 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0Oqp-0000000DLQ2-2mni for linux-arm-kernel@lists.infradead.org; Fri, 26 Apr 2024 16:55:09 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 86A68CE1CB0; Fri, 26 Apr 2024 16:55:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 50C9CC113CD; Fri, 26 Apr 2024 16:55:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714150504; bh=FgJb8ip6P9+AklWn3gKziPHPVzIvMpCQOIrfmY6RwzA=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=vA4G+RUS2HCJjpvDO8FpBD1l0DUJmIj8idLllzwfJsLdQt5/XX8873dkUW6c5xMI5 quEPkGlllVKbBheAhqwwkDJozCkwTrWYn6p2MO/WnKhnib7JQKC9ayvlAgw3EguLvy GuXmVKwv3RvIsafjIxleQBNT9As4DNDGlh4+3bb94zTQlCDTJuCAUknG05Nkqbrh9S DGkgEAQgTf5NazN86rOfIgfJBwtIaDibaEYlB7BCdxYzE0w+RL5h6X6zG7hC5oDA6c eUeW7IXR9DgBVL3m5ZhW/oaNbdbcYLObeIIrX6L6gDbTqOxPGOepvNHWII6c/hJgZO I1zf8zCgscUdQ== From: Puranjay Mohan To: Andrii Nakryiko Cc: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: Re: [PATCH bpf-next v3 1/2] arm64, bpf: add internal-only MOV instruction to resolve per-CPU addrs In-Reply-To: References: <20240426121349.97651-1-puranjay@kernel.org> <20240426121349.97651-2-puranjay@kernel.org> Date: Fri, 26 Apr 2024 16:55:01 +0000 Message-ID: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240426_095508_092208_92B43D54 X-CRM114-Status: GOOD ( 14.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org QW5kcmlpIE5ha3J5aWtvIDxhbmRyaWkubmFrcnlpa29AZ21haWwuY29tPiB3cml0ZXM6Cgo+IE9u IEZyaSwgQXByIDI2LCAyMDI0IGF0IDU6MTTigK9BTSBQdXJhbmpheSBNb2hhbiA8cHVyYW5qYXlA a2VybmVsLm9yZz4gd3JvdGU6Cj4+Cj4+IEZyb206IFB1cmFuamF5IE1vaGFuIDxwdXJhbmpheTEy QGdtYWlsLmNvbT4KPj4KPj4gU3VwcG9ydCBhbiBpbnN0cnVjdGlvbiBmb3IgcmVzb2x2aW5nIGFi c29sdXRlIGFkZHJlc3NlcyBvZiBwZXItQ1BVCj4+IGRhdGEgZnJvbSB0aGVpciBwZXItQ1BVIG9m ZnNldHMuIFRoaXMgaW5zdHJ1Y3Rpb24gaXMgaW50ZXJuYWwtb25seSBhbmQKPj4gdXNlcnMgYXJl IG5vdCBhbGxvd2VkIHRvIHVzZSB0aGVtIGRpcmVjdGx5LiBUaGV5IHdpbGwgb25seSBiZSB1c2Vk IGZvcgo+PiBpbnRlcm5hbCBpbmxpbmluZyBvcHRpbWl6YXRpb25zIGZvciBub3cgYmV0d2VlbiBC UEYgdmVyaWZpZXIgYW5kIEJQRgo+PiBKSVRzLgo+Pgo+PiBTaW5jZSBjb21taXQgNzE1ODYyNzY4 NmYwICgiYXJtNjQ6IHBlcmNwdTogaW1wbGVtZW50IG9wdGltaXNlZCBwY3B1Cj4+IGFjY2VzcyB1 c2luZyB0cGlkcl9lbDEiKSwgdGhlIHBlci1jcHUgb2Zmc2V0IGZvciB0aGUgQ1BVIGlzIHN0b3Jl ZCBpbgo+PiB0aGUgdHBpZHJfZWwxLzIgcmVnaXN0ZXIgb2YgdGhhdCBDUFUuCj4+Cj4+IFRvIHN1 cHBvcnQgdGhpcyBCUEYgaW5zdHJ1Y3Rpb24gaW4gdGhlIEFSTTY0IEpJVCwgdGhlIGZvbGxvd2lu ZyBBUk02NAo+PiBpbnN0cnVjdGlvbnMgYXJlIGVtaXR0ZWQ6Cj4+Cj4+IG1vdiBkc3QsIHNyYyAg ICAgICAgICAgIC8vIE1vdmUgc3JjIHRvIGRzdCwgaWYgc3JjICE9IGRzdAo+PiBtcnMgdG1wLCB0 cGlkcl9lbDEvMiAgICAvLyBNb3ZlIHBlci1jcHUgb2Zmc2V0IG9mIHRoZSBjdXJyZW50IGNwdSBp biB0bXAuCj4+IGFkZCBkc3QsIGRzdCwgdG1wICAgICAgIC8vIEFkZCB0aGUgcGVyIGNwdSBvZmZz ZXQgdG8gdGhlIGRzdC4KPj4KPj4gVG8gbWVhc3VyZSB0aGUgcGVyZm9ybWFuY2UgaW1wcm92ZW1l bnQgcHJvdmlkZWQgYnkgdGhpcyBjaGFuZ2UsIHRoZQo+PiBiZW5jaG1hcmsgaW4gWzFdIHdhcyB1 c2VkOgo+Pgo+PiBCZWZvcmU6Cj4+IGdsb2ItYXJyLWluYyAgIDogICAyMy41OTcgwrEgMC4wMTJN L3MKPj4gYXJyLWluYyAgICAgICAgOiAgIDIzLjE3MyDCsSAwLjAxOU0vcwo+PiBoYXNoLWluYyAg ICAgICA6ICAgMTIuMTg2IMKxIDAuMDI4TS9zCj4+Cj4+IEFmdGVyOgo+PiBnbG9iLWFyci1pbmMg ICA6ICAgMjMuODE5IMKxIDAuMDM0TS9zCj4+IGFyci1pbmMgICAgICAgIDogICAyMy4yODUgwrEg MC4wMTdNL3MKPgo+IEkgc3RpbGwgZXhwZWN0ZWQgYSBiZXR0ZXIgaW1wcm92ZW1lbnQgKGdsb2Jh bC1hcnItaW5jJ3MgcmVzdWx0cwo+IGltcHJvdmVkIG1vcmUgdGhhbiBhcnItaW5jLCB3aGljaCBp cyBjb21wbGV0ZWx5IGRpZmZlcmVudCBmcm9tCj4geDg2LTY0KSwgYnV0IGl0J3Mgc3RpbGwgYSBn b29kIHRoaW5nIHRvIHN1cHBvcnQgdGhpcyBmb3IgYXJtNjQsIG9mCj4gY291cnNlLgo+Cj4gYWNr IGZvciBnZW5lcmljIHBhcnRzIEkgY2FuIHVuZGVyc3RhbmQ6Cj4KPiBBY2tlZC1ieTogQW5kcmlp IE5ha3J5aWtvIDxhbmRyaWlAa2VybmVsLm9yZz4KPgoKSSB3aWxsIGhhdmUgdG8gZG8gbW9yZSBy ZXNlYXJjaCB0byBmaW5kIHdoeSB3ZSBkb24ndCBzZWUgdmVyeSBoaWdoCmltcHJvdmVtZW50LgoK QnV0IHRoaXMgaXMgd2hhdCBpcyBoYXBwZW5pbmcgaGVyZToKClRoaXMgd2FzIHRoZSBjb21wbGV0 ZSBwaWN0dXJlIGJlZm9yZSBpbmxpbmluZzoKCmludCBjcHUgPSBicGZfZ2V0X3NtcF9wcm9jZXNz b3JfaWQoKTsKbW92ICAgICB4MTAsICMweGZmZmZmZmZmZmZmZmQ0YTgKbW92ayAgICB4MTAsICMw eDgwMmMsIGxzbCAjMTYKbW92ayAgICB4MTAsICMweDgwMDAsIGxzbCAjMzIKYmxyICAgICB4MTAg LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tPiBub3AKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBub3AKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhZHJwICAgIHgwLCAweGZm ZmY4MDAwODIxMjgwMDAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBtcnMgICAgIHgxLCB0cGlkcl9lbDEKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhZGQgICAgIHgwLCB4MCwgIzB4OAogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxkcnN3ICAgeDAs IFt4MCwgeDFdCiAgICAgICAgICAgIDwtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tcmV0CmFkZCAgICAgeDcsIHgwLCAjMHgwCgoKTm93IHdlIGhhdmU6CgppbnQgY3B1ID0g YnBmX2dldF9zbXBfcHJvY2Vzc29yX2lkKCk7Cm1vdiAgICAgeDcsICMweGZmZmY4MDAwZmZmZmZm ZmYKbW92ayAgICB4NywgIzB4ODIxMiwgbHNsICMxNgptb3ZrICAgIHg3LCAjMHg4MDA4Cm1ycyAg ICAgeDEwLCB0cGlkcl9lbDEKYWRkICAgICB4NywgeDcsIHgxMApsZHIgICAgIHc3LCBbeDddCgoK U28sIHdlIGhhdmUgcmVtb3ZlZCBtdWx0aXBsZSBpbnN0cnVjdGlvbnMgaW5jbHVkaW5nIGEgYnJh bmNoIGFuZCBhCnJldHVybi4gSSB3YXMgZXhwZWN0aW5nIHRvIHNlZSBtb3JlIGltcHJvdmVtZW50 LiBUaGlzIGJlbmNobWFyayBpcyB0YWtlbgpmcm9tIGEgS1ZNIGJhc2VkIHZpcnR1YWwgbWFjaGlu ZSwgbWF5YmUgaWYgSSBkbyBpdCBvbiBiYXJlLW1ldGFsIEkgd291bGQKc2VlIG1vcmUgaW1wcm92 ZW1lbnQgPwoKVGhhbmtzLApQdXJhbmpheQoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJt LWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21h aWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=