From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64F16C4361B for ; Wed, 16 Dec 2020 22:07:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B460233A0 for ; Wed, 16 Dec 2020 22:07:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730134AbgLPWGo (ORCPT ); Wed, 16 Dec 2020 17:06:44 -0500 Received: from youngberry.canonical.com ([91.189.89.112]:60262 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730130AbgLPWGo (ORCPT ); Wed, 16 Dec 2020 17:06:44 -0500 Received: from mail-ot1-f71.google.com ([209.85.210.71]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1kpevp-0005JO-Bw for linux-kselftest@vger.kernel.org; Wed, 16 Dec 2020 22:06:01 +0000 Received: by mail-ot1-f71.google.com with SMTP id l67so11709753otc.10 for ; Wed, 16 Dec 2020 14:06:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=fGvoLcc7ZCHu79Q6fApxlF0Se+MpASUkI3Get7rgw7g=; b=cF2J/ypYE5gMmAnZt7p+uTydi0M0DhATVq5MjMFC/kWNWsqNJEubl7+dmFxZO439Ej gHEp6GsAJ5qblFTHdvqDX34K9iNSk5lMQ314UwYIhlF4b9uF7mqztKNjwYcvC5kHA8oZ X9kS3dJ1ECwN41vWDxUZ/w3Aw+JSjFiLZN5OplZXYyHvrpKIp37XufTbgA18AuSFpZmD xLhsS0Gks/zBxK0vUtm19UiqiS40hD7DcPU2VLT1TwQsswiL9OUl5Ym1zrrT+F/CNhwe k7kl6P5pJc8z61oKXbsfGsDs3ZszfPL6RnDA4A7HrmvH0lzsFgg4tERPfEtyNuWj/wr0 NXHg== X-Gm-Message-State: AOAM5333S4zVC3xJfjxLeHHuP6WKDVMTaFRLxqJ1Y494JJN07fTJWDCI 3aL6WRewwIVMuv/v7o3GKAa2N2O5FfxcncRky5uz81L86s9Cs1xCJIsVMnnn9KixX/HlQt7oD7I vCZStACcM8aASlKGKozFhQMjgPF8auoULvj4zxcBmdM/ntw== X-Received: by 2002:a9d:4588:: with SMTP id x8mr27874549ote.169.1608156360258; Wed, 16 Dec 2020 14:06:00 -0800 (PST) X-Google-Smtp-Source: ABdhPJwTj61vXgGhhnYgkOcQnHPGC/0/gTwApoGoxpWCunInbnH1FUNz6+hq5Ccq0G4yHGBmslW7JA== X-Received: by 2002:a9d:4588:: with SMTP id x8mr27874534ote.169.1608156360008; Wed, 16 Dec 2020 14:06:00 -0800 (PST) Received: from localhost ([2605:a601:ac0f:820:8f6:4890:1c98:9813]) by smtp.gmail.com with ESMTPSA id m10sm725721oig.27.2020.12.16.14.05.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 14:05:59 -0800 (PST) Date: Wed, 16 Dec 2020 16:05:58 -0600 From: Seth Forshee To: Mark Brown Cc: Shuah Khan , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Diaz , Veronika Kabatova , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Guillaume Tucker , Kevin Hilman Subject: Re: [PATCH] selftests: Skip BPF seftests by default Message-ID: References: <20201210185233.28091-1-broonie@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201210185233.28091-1-broonie@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org On Thu, Dec 10, 2020 at 06:52:33PM +0000, Mark Brown wrote: > The BPF selftests have build time dependencies on cutting edge versions > of tools in the BPF ecosystem including LLVM which are more involved > to satisfy than more typical requirements like installing a package from > your distribution. This causes issues for users looking at kselftest in > as a whole who find that a default build of kselftest fails and that > resolving this is time consuming and adds administrative overhead. The > fast pace of BPF development and the need for a full BPF stack to do > substantial development or validation work on the code mean that people > working directly on it don't see a reasonable way to keep supporting > older environments without causing problems with the usability of the > BPF tests in BPF development so these requirements are unlikely to be > relaxed in the immediate future. > > There is already support for skipping targets so in order to reduce the > barrier to entry for people interested in kselftest as a whole let's use > that to skip the BPF tests by default when people work with the top > level kselftest build system. Users can still build the BPF selftests > as part of the wider kselftest build by specifying SKIP_TARGETS, > including setting an empty SKIP_TARGETS to build everything. They can > also continue to build the BPF selftests individually in cases where > they are specifically focused on BPF. > > This isn't ideal since it means people will need to take special steps > to build the BPF tests but the dependencies mean that realistically this > is already the case to some extent and it makes it easier for people to > pick up and work with the other selftests which is hopefully a net win. > > Signed-off-by: Mark Brown Why not just remove the line which adds bpf to TARGETS? This has the same effect, but doesn't require an emtpy SKIP_TARGETS to run them. We have testing scripts which use 'make TARGETS=bpf ...' which will have to be updated, and I doubt we are the only ones. I also feel like this creates confusing semantics around SKIP_TARGETS. If I don't supply a value then I don't get the bpf selftests, but then if I try to use SKIP_TARGETS to skip some other test suddenly I do get them. That's counterintuitive. I also wanted to point out that the net/test_bpf.sh selftest requires having the test_bpf module from the bpf selftest build. So when the bpf selftests aren't built this test is guaranteed to fail. Though it would be nice if the net selftests didn't require building the bpf self tests in order to pass. Thanks, Seth > --- > tools/testing/selftests/Makefile | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile > index afbab4aeef3c..8a917cb4426a 100644 > --- a/tools/testing/selftests/Makefile > +++ b/tools/testing/selftests/Makefile > @@ -77,8 +77,10 @@ TARGETS += zram > TARGETS_HOTPLUG = cpu-hotplug > TARGETS_HOTPLUG += memory-hotplug > > -# User can optionally provide a TARGETS skiplist. > -SKIP_TARGETS ?= > +# User can optionally provide a TARGETS skiplist. By default we skip > +# BPF since it has cutting edge build time dependencies which require > +# more effort to install. > +SKIP_TARGETS ?= bpf > ifneq ($(SKIP_TARGETS),) > TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS)) > override TARGETS := $(TMP) > -- > 2.20.1 >