From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Monjalon Subject: Re: [PATCH v2] app/test: enhance sanity script Date: Sun, 20 May 2018 21:21:08 +0200 Message-ID: <15428018.4IKRziEG8p@xps> References: <1524835241-12778-1-git-send-email-reshma.pattan@intel.com> <12834353.uf7ZmsVvok@xps> <7AE31235A30B41498D1C31348DC858BD5B3D64D6@IRSMSX103.ger.corp.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7Bit Cc: "dev@dpdk.org" , "Burakov, Anatoly" , "Pattan, Reshma" To: "Parthasarathy, JananeeX M" Return-path: Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id 0ADB58D9D for ; Sun, 20 May 2018 21:21:12 +0200 (CEST) In-Reply-To: <7AE31235A30B41498D1C31348DC858BD5B3D64D6@IRSMSX103.ger.corp.intel.com> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" 20/05/2018 19:41, Parthasarathy, JananeeX M: > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > 18/05/2018 15:32, Parthasarathy, JananeeX M: > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > > 15/05/2018 18:14, Parthasarathy, JananeeX M: > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > > > > > > I think we need to keep a fast subset of tests. > > > > > > > > Please check in mk/rte.sdktest.mk: > > > > PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,ti > > > > mer_perf > > > > > > > > test-fast: BLACKLIST=-$(PERFLIST) > > > > > > We have added the below test cases to blacklist in mk/rte.sdktest.mk > > > reciprocal_division,reciprocal_division_perf,lpm_perf,red_all > > > > > > Now time taken is around 4mins. > > > Please let us know is there any minimum time limit for test execution. > > > Is this time duration fine? > > > > 4 minutes, it's a bit long for developer oriented tests. > > > > We need to distinguish which tests are enough for quick tests, > > and which ones can be in another category for full validation purpose. > > > > Please can you make a summary of which tests you are adding, > > and propose a priority classification. > > > > Thanks > > Please find below the test cases newly added to sanity script (autotest_data.py) with priority classification. > > P1 - Main test cases We can move some tests from P1. Examples: - bonding can be P2 with other drivers. - PMD perf and ring perf can P3 with other perf tests. > P2 - Cryptodev test cases which requires dependent libraries (Can be added if the environment is set accordingly) > P3 - Perf test cases and others which takes more time (> 10s) (these will be in the blacklist test cases) > P4 - Logging/Dump test cases (to be checked whether these are really necessary) I think this classification is good. We can adjust the categories in the makefile to reflect it. The goal is to have a really quick and core subset (equal P1 category), and other categories available with a different make command (drivers for P2, perf for P3 and debug for P4). > The blacklist is updated in makefile "rte.sdktest.mk" with below list > of test cases to reduce the total test execution time. > > Blacklist test cases are ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf, > reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,barrier, > hash_multiwriter,timer_racecond,efd,hash_functions,eventdev_selftest_sw, > member_perf,efd_perf,lpm6_perf,red_perf,distributor_perf > > Currently after the above blacklist update, the total execution time of "make test-fast" is 54s. Less than a minute is a good limit! Maybe we can do even better with above suggestions. Thanks for working on it. Feel free to send a new patch when you think it is ready. We can try to have the new autotests when starting 18.08. Then we should communicate to enforce a better coverage of the autotests.