From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90792CA0ECB for ; Mon, 11 Sep 2023 22:07:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356015AbjIKWCj (ORCPT ); Mon, 11 Sep 2023 18:02:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236280AbjIKKHq (ORCPT ); Mon, 11 Sep 2023 06:07:46 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 594E7101; Mon, 11 Sep 2023 03:07:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1694426862; x=1725962862; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=oh0Zy/V3VyMAu5In2Ipg/gMQQV8/i4gpEGPeqw7tFXI=; b=PUtX8EWJ7889ia3CxNJ0vCuyNykqr+UM54a895lj/V7+5QR8JB3kqUNE 2azyUzKXm4uqRMhEXb6ZjtbzJIi77CIdn98lFAQnMzISUYsdcvK1pKqmt ou8656UWlev5Vd44GL8qc9M5Uwqqva0Pi5xo5lQo2CZjO6qffSVGpWFQ4 yFAQIsHRcj+kw8JRPutNML4ERBCvGnFqZiow9lQjeQ1HihUFAgT4peW+9 Z5J0uqEv225BehkgqxA10dpnCzKfLs132UlZfW6+s6HSyJT8wUYh7VE+0 3IGxrn22OWP1OHCoP/3THv8UplvysQWss+Wev8JheJbfbKrr9iN1EfKbu g==; X-IronPort-AV: E=McAfee;i="6600,9927,10829"; a="357484290" X-IronPort-AV: E=Sophos;i="6.02,243,1688454000"; d="scan'208";a="357484290" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2023 03:07:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10829"; a="719938160" X-IronPort-AV: E=Sophos;i="6.02,243,1688454000"; d="scan'208";a="719938160" Received: from kschuele-mobl1.ger.corp.intel.com (HELO localhost) ([10.252.63.119]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2023 03:07:37 -0700 From: Jani Nikula To: Maxime Ripard , Brendan Higgins , David Gow , David Airlie , Daniel Vetter , Maarten Lankhorst , Thomas Zimmermann Cc: linux-kernel@vger.kernel.org, Maxime Ripard , =?utf-8?Q?Ma=C3=ADra?= Canal , dri-devel@lists.freedesktop.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com Subject: Re: [PATCH 1/2] kunit: Warn if tests are slow In-Reply-To: <20230911-kms-slow-tests-v1-1-d3800a69a1a1@kernel.org> Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo References: <20230911-kms-slow-tests-v1-0-d3800a69a1a1@kernel.org> <20230911-kms-slow-tests-v1-1-d3800a69a1a1@kernel.org> Date: Mon, 11 Sep 2023 13:07:35 +0300 Message-ID: <87leddf2fs.fsf@intel.com> MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 11 Sep 2023, Maxime Ripard wrote: > Kunit recently gained support to setup attributes, the first one being > the speed of a given test, then allowing to filter out slow tests. > > A slow test is defined in the documentation as taking more than one > second. There's an another speed attribute called "super slow" but whose > definition is less clear. > > Add support to the test runner to check the test execution time, and > report tests that should be marked as slow but aren't. > > Signed-off-by: Maxime Ripard > --- > lib/kunit/test.c | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c > index 49698a168437..a3b924501f3d 100644 > --- a/lib/kunit/test.c > +++ b/lib/kunit/test.c > @@ -379,6 +379,9 @@ static void kunit_run_case_internal(struct kunit *test, > struct kunit_suite *suite, > struct kunit_case *test_case) > { > + struct timespec64 start, end; > + struct timespec64 duration; > + > if (suite->init) { > int ret; > > @@ -390,7 +393,20 @@ static void kunit_run_case_internal(struct kunit *test, > } > } > > + ktime_get_ts64(&start); > + > test_case->run_case(test); > + > + ktime_get_ts64(&end); > + > + duration = timespec64_sub(end, start); > + > + if (duration.tv_sec >= 1 && > + (test_case->attr.speed == KUNIT_SPEED_UNSET || > + test_case->attr.speed >= KUNIT_SPEED_NORMAL)) > + kunit_warn(test, > + "Test should be marked slow (runtime: %lld.%09lds)", > + duration.tv_sec, duration.tv_nsec); Two thoughts: Should there be some tolerance here? Otherwise we're flagging this on the slowest machines, and we'll be defining tests slow based on that. Like, warn if it takes more than 2 seconds. What if someone makes a test faster, but forgets to update the attribute? Should we also flag slow tests that are in fact fast? BR, Jani. > } > > static void kunit_case_internal_cleanup(struct kunit *test) -- Jani Nikula, Intel Open Source Graphics Center