From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14326C433DF for ; Fri, 7 Aug 2020 17:25:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B7C93221E5 for ; Fri, 7 Aug 2020 17:25:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="rXo3tXM8"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="RburP085" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7C93221E5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=4TSg4NXNcafVvJj3aZMWmZFAwcSWytQ6pGVARLfNKVQ=; b=rXo3tXM8qK70UIXMf3Pflg/Yn TbwJG71UDqSJrOb3ldMP5uypEDpZ/7vmemQ8CFdflDmVeh+jFAqt4vpPiYb8R8WdCgeeNkRusBFqz RCk+OmksNvmPKiqWAcEVDlFgvW4hPhxSseroIJ5Eev/VfK2Az/mVLARH+lFcfgJacBYt0d1nN5q1g Zd3CQ1vFy7H9RJJmKpAw6bcfH1r3Mvb1LFyYj4P9GMVsRAZ6wDf150Nzn8syHh5nZHLUdddAQeW6A 5SMk2k1/+AdQsSn+FnQeKyAP0gOPnT5J8naQ02i5ZJmcHf/SXY+TUqlwD8NN1H97QtvXE4B6UpiW/ lAVSeGZsQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k466O-0001eP-CP; Fri, 07 Aug 2020 17:24:20 +0000 Received: from mail-ej1-x632.google.com ([2a00:1450:4864:20::632]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k466L-0001dA-MM for linux-arm-kernel@lists.infradead.org; Fri, 07 Aug 2020 17:24:18 +0000 Received: by mail-ej1-x632.google.com with SMTP id bo3so2839131ejb.11 for ; Fri, 07 Aug 2020 10:24:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=eyQgcKKa6sa8KC5xVdsdC/+RP8gsTOEmgKv3cyYA1MY=; b=RburP0850JrB1ImGEcqvCCrS0FPCzgg8OzjYrP/lLOpPt1c790kxesPUc0C7v69Ey2 sFMmbGQMqDnvOSBxAOOOVxWaS5++MIiU1Jxf6OFavQ6CeA5LqnXlsAGMGfuY1dvlKE/X 534vpIRsy9LmEDu8kLzHSCXZtuBUtOyLxEd7/L+/emEA6vu4lLPEg4W6Zq1cnNtWqN+z qnsD8iaqmn2lwh7OXOeeqRFKk/BRLoj+oOwcQ0Nh6ziqePbnbszAJEIfYzoZ1QS1GB1P gTDvuxe42jwAE2DaRfe8WHAkaD+JsUxewtxx6asZsFf7Qcd1s5BKe7tntPNm6aVK5SVR KnZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=eyQgcKKa6sa8KC5xVdsdC/+RP8gsTOEmgKv3cyYA1MY=; b=f4v5Ae9jgvt2gr/vIVp/Bnmsjm7Fqxl8WeM5tIU0hWQSrsikzDBeKjFQ5I4zPHoB+u NBL0xxH7ulau77msFMvW/jOAnBiGFG3lFUp5h76uHz4qXxsyZT5FFgyK0gmUvc1Mo2EU b+a7PUgvSofgpf298u8UaLbNDRPeFBAGSiQbRA7dxdSSRpHs+4QGsqPlSDIRHN9xdhjD PXYld4Utv9QTRGo9jjJcq6CwXISyBkrGSz5ETaE5Wvb2tI5MfpttObWpkOTtYQWIAryF LcQ8WyKMR/ecDWR5MEXdYgtsFVQcCYwPDy3DfKsSlIvbvZqV8Xbswd9BwMoh3/vvBAiE m2zg== X-Gm-Message-State: AOAM53023G/ckAjPOZre+XmSQmZbvOthrdoNsEE3ZALRtoqKQOfeFw+F 9vw1fKu9DShjFRBiTvJgv1rHvQ== X-Google-Smtp-Source: ABdhPJwd0kaCd8LoGMztjNARsWWNmMLRIR5iHEiPUMWU+JMxTyu36tKm/5HGQGV58Ickq2qC8g5W5w== X-Received: by 2002:a17:906:3cc:: with SMTP id c12mr9967342eja.222.1596821049838; Fri, 07 Aug 2020 10:24:09 -0700 (PDT) Received: from myrica ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id e8sm5704686edy.68.2020.08.07.10.24.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Aug 2020 10:24:09 -0700 (PDT) Date: Fri, 7 Aug 2020 19:23:53 +0200 From: Jean-Philippe Brucker To: Jakov Petrina Subject: Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms Message-ID: <20200807172353.GA624812@myrica> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200807_132417_945748_2A0F1A5D X-CRM114-Status: GOOD ( 28.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Luka Perkov , Juraj Vijtiuk , Jakov Smolic , bpf@vger.kernel.org, Andrii Nakryiko , Andrii Nakryiko , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, [Adding the linux-arm-kernel list on Cc] On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote: > Hi everyone, > > recently we have begun extensive research into eBPF and related > technologies. Seeking an easier development process, we have switched over > to using the eBPF CO-RE [0] approach internally which has enabled us to > simplify most aspects of eBPF development, especially those related to > cross-compilation. > > However, as part of these efforts we have stumbled upon several problems > that we feel would benefit from a community discussion where we may share > our solutions and discuss alternatives moving forward. > > As a reference point, we have started researching and modifying several eBPF > CO-RE samples that have been developed or migrated from existing `bcc` > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools` > directory [1]. Some of these samples have just recently been converted to > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has > proven to be very interesting. > > First showstopper for cross-compiling aforementioned example on the ARM > 32-bit platform has been with regards to generation of the required > `vmlinux.h` kernel header from the BTF information. More specifically, our > initial approach to have e.g. a compilation target dependency which would > invoke `bpftool` at configure time was not appropriate due to several > issues: a) CO-RE requires host kernel to have been compiled in such a way to > expose BTF information which may not available, and b) the generated > `vmlinux.h` was actually architecture-specific. > > The second point proved interesting because `tcpconnect` makes use of the > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as > the first function parameter. The `pt_regs` structure is defined by the > kernel and is architecture-specific. Since `libbpf` does have > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h` > resulted in cross-compilation failure as `libbpf` provided macros that work > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To > resolve this issue, we have resorted to including pre-generated > `_vmlinux.h` files in our CO-RE build system. > > However, there are certainly drawbacks to this approach: a) (relatively) > large file size of the generated headers, b) regular maintenance to > re-generate the header files for various architectures and kernel versions, > and c) incompatible definitions being generated, to name a few. This last > point relates to the the fact that our `aarch64`/`arm64` kernel generates > the following definition using `bpftool`, which has resulted in compilation > failure: > > ``` > typedef __Poly8_t poly8x16_t[16]; > ``` > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We > have opted to comment out this line as there was no additional `poly8x16_t` > usage in the header file. It looks like this "__Poly8_t" type is internal to GCC (provided in arm_neon.h) and clang has its own internals. I managed to reproduce this with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at the moment. Maybe libbpf should generate defines to translate these intrinsics between clang and gcc? Not very elegant. I'll take another look next week. > Given various issues we have encountered so far (among which is a kernel > panic/crash on a specific device), additional input and feedback regarding > cross-compilation of the eBPF utilities would be greatly appreciated. I don't know if there is a room for improvement regarding your a) and b) points, as I think the added complexity is inherent to cross-building. But kernel crashes definitely need to be fixed, as well as the above problem. Thanks, Jean _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel