From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BF7BC6FD1D for ; Tue, 4 Apr 2023 12:48:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED15E6B0071; Tue, 4 Apr 2023 08:48:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E81786B0072; Tue, 4 Apr 2023 08:48:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D48E16B0074; Tue, 4 Apr 2023 08:48:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C57526B0071 for ; Tue, 4 Apr 2023 08:48:57 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4643380AE5 for ; Tue, 4 Apr 2023 12:48:57 +0000 (UTC) X-FDA: 80643688314.07.FB1FDAD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 838E31A0015 for ; Tue, 4 Apr 2023 12:48:53 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JF0uKUnS; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680612534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i3qiT0TK0SRhIz1YZD0OkKsTfa99dtrpFH3zJu9LCi8=; b=dDhEwFcbboK3Rn813kPthiNUN4cCTPt4AkrbcuBIYUYctDP2bT7xnK39PFQu5zqt+pvZXR K6s24AHiJg33AXJYELyiOL/9BhRS3GIMQFSiU7MeGbkqfnw5r6J6DFB8kLJNCwl9z/uQd8 2qieVx7JW8vl+XmgHDKQNFEwzS6R14M= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JF0uKUnS; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680612534; a=rsa-sha256; cv=none; b=Qc/ovk3Op8uk3oNuR22K+6yTzGNA7SPCdikZD4ePtemEPcJyVAgpMz7jGkiPggLFEd1EvA 5KnQgG6BVzC4AhRLFAJRh8PeiXXlcm6VoUxCmrJzztXVknEcl3yA0i3nd1UQo7CmWqzCXX A7tsD+lDEnbJ4LBahekHsE/ytiY5Szk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680612532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i3qiT0TK0SRhIz1YZD0OkKsTfa99dtrpFH3zJu9LCi8=; b=JF0uKUnSyi1p9HKd7a7UYXXpAdCtFD0WaFR3ZN7JbuGAqGSXMIWd2OPCi6tofJcBFJABDH wjmxR4g2neCv9/pynPoSR9eV9sJibYcX8XKvglqywVVixAmO12sxwa2tI8gQPN0gDGl6Km 9FVB+NrlB3Xs6EIiK4M+YlWMh+ybk3Y= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-158-FT0b5c_pNYWLnj1ZrOzsOw-1; Tue, 04 Apr 2023 08:48:51 -0400 X-MC-Unique: FT0b5c_pNYWLnj1ZrOzsOw-1 Received: by mail-wr1-f71.google.com with SMTP id k16-20020adfd230000000b002cfe7555486so3685630wrh.13 for ; Tue, 04 Apr 2023 05:48:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612530; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=i3qiT0TK0SRhIz1YZD0OkKsTfa99dtrpFH3zJu9LCi8=; b=HfeCWtXR8lyFruQh4qBQMWF1c0J/UU60FTCjpTfKTZj0KJOguPjY8IPm7UdyGXb2aw Zuef8B1+aoWvsoERvYqj62LR8Tl2V2SdyYWg/Rl6e62P6jCptPVvB4irDEU86Q7SVR7q vvZ5U212/lTSiN//zoHTA07asSCt1Nx2Qb6d352q7zVhvewHtCNcOnKRPsGKNEy+0q9C D7lJShhL4yzYeBMxGDmWq8jNCHWWtlFwV/4ZIxIlwJ/oewAFdm6OBTk2zeCl6KO2Ecg2 M91z58zI1Mapv/7R572GN2v4ol8BpoXrzvmk+BAE9QpGIKx9u01A+C7+SYgLhwtcLidQ /Fpg== X-Gm-Message-State: AAQBX9f9n6gVkDxLLqobUnrK6aGusXwXiIUc+QO6/cCInm5JwrStZmgf 1NfimAvg43aZGksVOCqdfDnB+IxxGUXtdURCx+AuW/g719CpH2rZfznQCnJc07vp6dE6E5JjFsA r7xiCHJnTg1k= X-Received: by 2002:a5d:634b:0:b0:2cf:ef01:1def with SMTP id b11-20020a5d634b000000b002cfef011defmr1375952wrw.30.1680612530225; Tue, 04 Apr 2023 05:48:50 -0700 (PDT) X-Google-Smtp-Source: AKy350YtLckYOuIk4nDtcNSZYcM/mUTR2i0YeiImBkQwV0Kjcdo6mXo+2IEc+GWC5ARjj0yV2RGzaQ== X-Received: by 2002:a5d:634b:0:b0:2cf:ef01:1def with SMTP id b11-20020a5d634b000000b002cfef011defmr1375936wrw.30.1680612529861; Tue, 04 Apr 2023 05:48:49 -0700 (PDT) Received: from ?IPV6:2003:cb:c709:b600:e63:6c3b:7b5d:f439? (p200300cbc709b6000e636c3b7b5df439.dip0.t-ipconnect.de. [2003:cb:c709:b600:e63:6c3b:7b5d:f439]) by smtp.gmail.com with ESMTPSA id o15-20020a5d474f000000b002d7a75a2c20sm12289754wrs.80.2023.04.04.05.48.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 04 Apr 2023 05:48:48 -0700 (PDT) Message-ID: Date: Tue, 4 Apr 2023 14:48:47 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 To: Peter Xu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrea Arcangeli , Andrew Morton , Nadav Amit , Mike Kravetz , Axel Rasmussen , Leonardo Bras Soares Passos , Mike Rapoport References: <20230330155707.3106228-1-peterx@redhat.com> <20230330160752.3107283-1-peterx@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH 16/29] selftests/mm: UFFDIO_API test In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 838E31A0015 X-Rspam-User: X-Stat-Signature: yk88z7pj4orwqfx6x64ksjcyy9q3kmja X-HE-Tag: 1680612533-805992 X-HE-Meta: U2FsdGVkX1/9bLWPL4XL3YF0lFYe0hg4Z+nW2ahxgaXaGKra9FJUh+OrmCHtUFmRMRa+6myhc5PxhacbBiZSgNpGqMc5NxEN+beJvRbeGGUL0+0QPwbST6QF1eKeb/LD8T2CAKbv1kpM8OgE9kNPjjgrL0ECoOfGUPxJhMUHUVpYMILxZ6r8wLrXtcWsXSm1QEjpUW64oJvgtnvBPJcRfMy4kAWaTDfeABWo262ZQgymJnC/cpRkNpnYpcFNVKLHf0BeuolkxQN4dNRjsOXw7g3CBxoEGk90nG0Pf/H4MP+7G02y78D+lnr0qix8pLI72FXbjLDK8hgFcpnuGj+x+wQ7GiyhJlAj4ZK9L63z03YqNV1gwUTLi1CsqxnR025oRCjpvUwDVYSlVH+NE8LLv8bWYf6FvB/bKdUnxlRPfuTYAG9RVWWVECGIBAChW7nOy6IA5XJDFSGTRuXs9PEK4RzJCnbnA6OnFE6eGo5JH2OoLGYDHItFBiECJDNaMamu2xSESSDHs4fiS/P7D5n+RkVgX2tUbWtZmDbyBtOAFN+Fd0iW++K0XoAh/Siicwf09bO/mUbMc77tJE6oX4BAmb9CMU3DWtblKXRUAfFbaN602BKcP9qIHcr6130Qj3GTFDBSO1JWXgEgSE3leVUyCKqkysdrKqk9NqY5WP3kaJPi2rEhq1ewhG/940rfZYkcqrZj4zHDQeekY3AGWP+gCRbHx+WcQrUY5CXY6LdWkEdDLtuPWDLWqDbPIpxT0rofRmElEwR1Y4PqRwDkj6HunQKdgpThgyarvNQUm9pu1R3fi7LVBgWCht66jbZr31DMqonYCngUUxNpYQyTNuLW99iismmztNX4MGqxaNQBGPlafqSCG0i/7FRY8LiL9uQcow6+uJpezO3obsMCmqtYEt7Gu1xx4lz9Mi4x4yqbX35iv0WvUmkCfEOI7dcMg2XIoxvgiTgpfn7ekjRktJu cdq/otNs 1c02dILmCWKlbQVgcf9jAZN4RdiqHC1cw1gCuyMv8IEG/nTVYPxQtwGTTOrPSo83BSdwm2T617ZJcdL8H8sV+/60VMjXWux89NvOF+wvOu5Y9A8To5+nDkPk8d+Im9d7EpT4Bc0fSbzHy20FbwYWpXMn/ySiy+bELC/MLQA6GOnyb5P/2gQumaecD2DgGEBlDGlL09vU301nxSM5Hds2E/OU6/RWn/4M6o5Mux4jvrCrOD4Jsl8yQAV/0d4DrfKIXQ6mNvSRIQz4sgStCyqIxRubSdc3Th/DGAK6rljU98IifA9GvbSaT1rYTZ1I7/qys2U5dZeSncrC9SeqLVvv2+fIbH4kpr1QG+Z/YJA/XpdolGcjEQFEimyB+oYDDNyNsSK4K/DxwvXlZnTgoxPP0mNQL2KOhKrTklLyaffrSrz9njVgFjy+QT0ge1ag9pu5YzLq32EDJeiIO8WS222QXDMuLySXVB+LeSH7zJZOHu3gAcAp1wk1AqmJHJ5SyMOYZt8HAhmoIl5mWDzY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000576, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 03.04.23 22:24, Peter Xu wrote: > On Mon, Apr 03, 2023 at 09:06:26PM +0200, David Hildenbrand wrote: >> On 03.04.23 18:43, Peter Xu wrote: >>> On Mon, Apr 03, 2023 at 09:59:50AM +0200, David Hildenbrand wrote: >>>> There is ksft_print_msg, ksft_test_result, ksft_test_result_fail, ... do we >>>> maybe want to convert properly to ksft while already at it? >>> >>> Yes, I started with trying to use that but found that there're not a lot of >>> things that I can leverage. >>> >>> Starting with ksft_set_plan() - I think this is something we call first. I >>> want the current unit test to skip everything if UFFD API test failed here, >>> then I need to feed in a dynamic number of "plan" into ksft_set_plan(). >>> But I never know after I ran the 1st test.. >> >> In cow.c I did that. Getting the number of tests right can be challenging >> indeed. > > IMHO the major thing is not about not easy to set, it's about there's > merely no benefit I can see of having that calculated at the start of a > test. Thinking about it, I believe I figured out why it makes sense. By specifying upfront how many tests you intend to run, the framework can check if you have the right number of pass/fail/skip during that test case execution. This requires a different way of writing tests: each test case is supposed to trigger the exact same number of pass/fail/skip on each possible path. If the numbers don't add up, it could indicate a possible bug in your tests. For example, not triggering a fail on some exit path. While your test execution might indicate "success", there is actually a hidden issue in your test. I started using the framework for all new tests, because I think it's quite nice and at least things will be a bit consistent and possibly tests easier to maintain. Having that said, I can understand why one might not want to use it. And that there are some things in there that might be improved. > > There's one thing it can do, that is when calling ksft_finished() it can be > used to know whether all tests are run, but sadly here we're calculating > everything just to make it match.. so it loses its last purpose.. IMHO. > >> >> Basic "feature availability" checks would go first (is uffd even around?), >> and depending on that you can set the plan. >> >> For everything else, you can skip instead of test, so it will still be >> accounted towards the plan. >> >>> >>> I can call ksft_set_plan() later than this, but it misses a few tests which >>> also looks weird. >> >> Yeah, it would be nice to simply make ksft_set_plan() optional. For example, >> make ksft_print_cnts() skip the comparison if ksft_plan == 0. At least >> ksft_exit_skip() handles that already in a descend way (below). >> >>> >>> It also seems to not really help anything at all and not obvious to use. >>> E.g. ksft_finished() will reference ksft_plan then it'll trigger >>> ksft_exit_fail() but here I want to make it SKIP if the 1st test failed >>> simply because the kernel probably doesn't have CONFIG_USERFAULTFD. >> >> You'd simply do that availability check first and then use ksft_exit_skip() >> in case not available I guess. >> >>> >>> Another example: I never figured what does x{fail|pass|skip} meant in the >>> header.. e.g. ksft_inc_xfail_cnt() is used nowhere so I cannot reference >>> either. Then I don't know when I should increase them. >> >> In cow.c I have the following flow: >> >> ksft_print_header(); >> ksft_set_plan(); >> ... tests ... >> err = ksft_get_fail_cnt(); >> if (err) >> ksft_exit_fail_msg(); >> return ksft_exit_pass(); >> >> That gives me: >> >> # [INFO] detected THP size: 2048 KiB >> # [INFO] detected hugetlb size: 2048 KiB >> # [INFO] detected hugetlb size: 1048576 KiB >> # [INFO] huge zeropage is enabled >> TAP version 13 >> 1..190 >> ... >> # Totals: pass:87 fail:0 xfail:0 xpass:0 skip:103 error:0 >> >> >> I didn't use xfail or xpass so far, but what I understood is that these are >> "expected failures" and "expected passes". fail/pass/skip are straight > > Yes, xfail can be expressed that way, but maybe not xpass? Otherwise it's > hard to identify what's the difference between xpass and pass, because IIUC > pass also means "expected to pass". > I'm a simple man, I use pass/fail/skip in my tests. But let's try figuring that out; a quick internet search (no idea how trustworthy) tells me that I was wrong about xpass: xfail: expected failure xpass: unexpected pass it essentially is: xfail -> pass xpass -> fail but with a slight semantic difference when thinking about a test case: ret = mmap(0, 0 ...); if (ret == MAP_FAILED) { XFAIL(); } else { XPASS(); } vs. ret = mmap(0, 0 ...); if (ret == MAP_FAILED) { PASS(); } else { FAIL(); } It's all inherited from other testing frameworks I think. And xpass seems to be completely unused and xfail mostly unused. So I wouldn't worry about that and simply use pass/fail/skip. >> forward. >> ksft_test_result_fail()/ksft_test_result_pass()/ksft_test_result_skip() are >> used to set them. >> >> You'd do availability checks before ksft_set_plan() and fail with a >> ksft_exit_skip() if the kernel doesn't support it. Then, you'd just use >> ksft_test_result_fail()/ksft_test_result_pass()/ksft_test_result_skip(). >> >>> >>> In short, to make the unit test behave as expected, I figured I'll just >>> write these few helpers and that's good enough for this unit test. That >>> takes perhaps 5 min anyway and isn't hugely bad for an unit test. >>> >>> Then I keep the exit code matching kselftests (KSFT_SKIP, etc.). >>> >>> What I can do here, though, is at least reuse the counters, e.g: >>> >>> ksft_inc_pass_cnt() / ksft_inc_fail_cnt() >>> >>> There's no ksft_inc_skip_cnt() so, maybe, I can just reuse >>> ksft_inc_xskip_cnt() assuming that counts "skip"s? >>> >>> Let me know if you have better ideas, I'll be happy to switch in that case. >> >> I guess once you start manually increasing/decreasing the cnt, you might be >> abusing the ksft framework indeed and are better off handling it differently >> :D > > I'm serious considering that to address your comment here, to show that I'm > trying my best to use whatever can help in this test case. :) Here reusing :) appreciated, but don't let my comments distract you. If you don't think ksft is a good fit (or any good), then don't use it. -- Thanks, David / dhildenb