From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oa1-f49.google.com (mail-oa1-f49.google.com [209.85.160.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B672A622 for ; Mon, 27 Mar 2023 02:44:32 +0000 (UTC) Received: by mail-oa1-f49.google.com with SMTP id 586e51a60fabf-17aceccdcf6so7814637fac.9 for ; Sun, 26 Mar 2023 19:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679885071; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=/IehCeOspUlraWZ1AW390EFHJWuA8vsflh6eGOs0IcA=; b=BbckQad35hdIB9AdNC5ttGTvQW6Ag3M3+i/mD07LFEO0/0OG5KtWfja/NltAxlqblv W92Ozr9ODRVfHcOKuSZFQOVApstuOaJaf1jfJ2Lb64GtA5L8Zs8Ncwz+ZzsvXcEaf9xQ 1O9LmIc5OyhrV6BFndIQzeXEsoOw4wst4uSqrcBB2hyi3vtYpe9ZVlgK3kkFMoItAtXn nvx6hdyvAmpXTM8c/4eqqzuqC/nQAqSoRHvc+LVCw5YsKeqfUwGZJhDPbPqkdim9W6RY 3/3eA9eY7BTV6QZ1PpgZ6vQXSiVkIwJhgicoiU2WbmnsJ1Uwc4U1WbcczpXFcXMPXnzw 8otg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679885071; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/IehCeOspUlraWZ1AW390EFHJWuA8vsflh6eGOs0IcA=; b=fuaUWfLe9WFzsXPv7Fft1htwCXz8VOdqUbbhoUP7dskyJX1QzndTvZMt97x+QB5E36 SuVu+FSHUxARCGr5o9xwckZDN14d9FaJOP6Gk1QAoFDzo+YDDJ6+bOPWjD6gXKv8JtUt Ydw2zIiGbh29G0MraF1VqDDu4dYCgY1BRHn8Zx2vWes8V4TP2q7Brq/sfgJxUazBv6YC 6+xhJZt2PNDve7nq4aL5zrGn4l4dAY6NUJGCZYs+I75RSXJ/IwEOoOqwGScaUDVZaxey 4v5ZNvxfP483+2onCDcYR+ALhAZ/NGdbWKt/O0/4ZHjlcwzOEqHEgyXeMzJAlXoNLmD1 1ctg== X-Gm-Message-State: AO0yUKX75uBSIYchSp3xxoq27JAH0oo8i9y98xS4m3wsD2Mb3RhaBld3 Wedcp47XeIb3ivQwRI5SaF8= X-Google-Smtp-Source: AKy350bKgFENzE6qQyN7mlnvfyBQfZAEoRGTFjt+++dpuLKoa54K7yO5TvsNKRANtW4od3i3j1/FfQ== X-Received: by 2002:a05:6870:3396:b0:17e:8ac6:5399 with SMTP id w22-20020a056870339600b0017e8ac65399mr7726353oae.0.1679885071798; Sun, 26 Mar 2023 19:44:31 -0700 (PDT) Received: from ?IPV6:2600:1700:2442:6db0:988b:7dda:764e:c744? ([2600:1700:2442:6db0:988b:7dda:764e:c744]) by smtp.gmail.com with ESMTPSA id vh22-20020a0568710d1600b00177c21c3ae1sm9429423oab.54.2023.03.26.19.44.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 26 Mar 2023 19:44:31 -0700 (PDT) Message-ID: <3a4ed4a0-5694-ba59-a141-26e37576ff67@gmail.com> Date: Sun, 26 Mar 2023 21:44:30 -0500 Precedence: bulk X-Mailing-List: kernelci@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: [KTAP V2 PATCH] ktap_v2: add recognized test name line Content-Language: en-US To: Daniel Latypov , Rae Moar Cc: davidgow@google.com, skhan@linuxfoundation.org, keescook@chromium.org, Tim.Bird@sony.com, brendanhiggins@google.com, corbet@lwn.net, guillaume.tucker@collabora.com, kernelci@lists.linux.dev, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org References: <20230316225915.494688-1-rmoar@google.com> From: Frank Rowand In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 3/20/23 14:21, Daniel Latypov wrote: > On Thu, Mar 16, 2023 at 3:59 PM Rae Moar wrote: >> >> Add recognition of the test name line ("# Subtest: ") to the KTAP v2 >> spec. >> >> The purpose of this line is to declare the name of a test before its >> results. This functionality is especially useful when trying to parse test >> results incrementally and when interpretting results after a crash. >> >> This line is already compliant with KTAP v1 as it is interpretted as a > > minor nit for if there's a v2, s/interprett/interpret (here and above) > > Also, I want to elaborate on the previous paragraph a bit more, in > case the motivation isn't clear. > The problem with TAP and KTAP as-is is that the name of a test case is > only known *after* it completes. > > So the scenario being referred to is > > KTAP version 1 > 1..1 > > > > It would be nice if parsers could report "test FOO caused a crash" as > opposed to "the first test case caused a crash, good luck figuring out > which one that was" > > Daniel It would be useful to take some of the motivation explanation from TAP14, which says (everything to the end of this email): Commented Subtests are encouraged, as they provide the following benefits: Easier for humans to read. For example: TAP version 14 1..1 ok 1 - hmm, what level is this? vs: TAP version 14 # Subtest: level 1 # Subtest: level 2 # Subtest: level 3 1..1 ok 1 - clearly level 3 Additional strictness around matching the Test Point description to Subtest Name can catch errors and detect accidentally interleaved output.