From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by yocto-www.yoctoproject.org (Postfix, from userid 118) id 29768E00A26; Thu, 6 Aug 2015 14:23:16 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on yocto-www.yoctoproject.org X-Spam-Level: X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI, TO_NO_BRKTS_PCNT autolearn=ham version=3.3.1 X-Spam-HAM-Report: * -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high * trust * [192.55.52.88 listed in list.dnswl.org] * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.5 TO_NO_BRKTS_PCNT To: misformatted + percentage Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id A9701E009B8 for ; Thu, 6 Aug 2015 14:23:12 -0700 (PDT) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga101.fm.intel.com with ESMTP; 06 Aug 2015 14:23:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.15,625,1432623600"; d="scan'208";a="763555898" Received: from linux.intel.com ([10.23.219.25]) by fmsmga001.fm.intel.com with ESMTP; 06 Aug 2015 14:23:12 -0700 Received: from linux.intel.com (vmed.fi.intel.com [10.237.72.65]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by linux.intel.com (Postfix) with ESMTP id B42EE6A4083 for ; Thu, 6 Aug 2015 14:22:20 -0700 (PDT) Date: Fri, 7 Aug 2015 00:23:02 +0300 From: Ed Bartosh To: toaster@yoctoproject.org Message-ID: <20150806212302.GA22787@linux.intel.com> MIME-Version: 1.0 Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Toaster testing X-BeenThere: toaster@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list Reply-To: ed.bartosh@linux.intel.com List-Id: Web based interface for BitBake List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 21:23:16 -0000 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, Recently I've been fixing toaster testing and discovered some info which I'd like to share with you. Toaster has several types of tests. I'll be talking here about Django testing. You can read about it in details in Django documentation: https://docs.djangoproject.com/en/1.8/topics/testing/overview/ Here is how it works for Toaster: $ cd bitbake/lib/toaster/ $ ./manage.py test toastergui Creating test database for alias 'default'... ....... ---------------------------------------------------------------------- Ran 7 tests in 0.256s OK toastergui is one of 3 Toaster modules with tests. The rest two are orm and bldcontrol. orm and toastergui tests are in working state and bldcontrol is still broken. It's possible to run both working tests. It can be done this way: $ ./manage.py test toastergui orm toastergui To see more verbose output you can use -v2 or -v3 commandline options: $ ./manage.py test -v2 toastergui Creating test database for alias 'default' (':memory:')... Creating tables ... Creating table auth_permission Creating table auth_group_permissions Creating table auth_group ... Creating table orm_logmessage Installing custom SQL ... Installing indexes ... Installed 0 object(s) from 0 fixture(s) test_get_base_call_returns_html (toaster.toastergui.tests.ViewTests) Basic test for all-projects view. ... ok test_get_json_call_returns_json (toaster.toastergui.tests.ViewTests) Test for all projects output in json format. ... ok test_xhr_createcustomrecipe_absent_parameter (toaster.toastergui.tests.ViewTests) Test not passing parameter to xhr_createcustomrecipe. ... ok test_xhr_createcustomrecipe_invalid_base_recipe (toaster.toastergui.tests.ViewTests) Test passing wrong base recipe id. ... ok test_xhr_createcustomrecipe_invalid_project (toaster.toastergui.tests.ViewTests) Test passing wrong project id. ... ok test_xhr_createcustomrecipe_ok (toaster.toastergui.tests.ViewTests) Test successful return from ReST API xhr_createcustomrecipe. ... ok test_xhr_datatypeahead_layer (toaster.toastergui.tests.ViewTests) Test ReST API xhr_datatypeahead. ... ok ---------------------------------------------------------------------- Ran 7 tests in 0.253s OK It's also possible to generate code coverage report using coverage.py tool. You can install in with pip or package manager of your Linux distro. I'm using OpenSUSE 13.2 and installed it this way: $ zypper install python-coverage To information tests should be run the way described in this document: https://docs.djangoproject.com/en/1.8/topics/testing/advanced/#integration-with-coverage-py For example, coverage report for toastergui tests can be generated the folowing way: $ coverage run --source ./toastergui/ ./manage.py test toastergui ... $ coverage report Name Stmts Miss Cover --------------------------------------------------------- toastergui/__init__ 0 0 100% toastergui/tables 192 143 26% toastergui/templatetags/__init__ 0 0 100% toastergui/templatetags/projecttags 153 108 29% toastergui/tests 77 0 100% toastergui/urls 5 0 100% toastergui/views 1399 1100 21% toastergui/widgets 169 134 21% --------------------------------------------------------- TOTAL 1995 1485 26% coverage report -m will include information about code lines not covered by tests. This is very useful option for test writers. I'd like to encaurage Toaster developers and contributors to run tests before sending patches for review to this mailing list and to upstream. It would be also great to cover more code with tests. I'd propose to concentrate on writing tests for toastergui module as this module is closer to end user UI than 2 others. PS: My fixes are not yet upstreamed. You can find them in poky-contrib branches ed/toaster/fix-orm-tests and ed/toaster/fix-toastergui-tests -- Regards, Ed