From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f74.google.com (mail-dl1-f74.google.com [74.125.82.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE3143CAA49 for ; Wed, 13 May 2026 23:05:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778713519; cv=none; b=YnMU6Ztq+5oeHRwbQ/NN63uJEy2VeEqieVtXiiq6fsJvlJx+pwgHrO09r5R1UaWLYYXXtW1aOxHIXQ+bVjFM6OtzdFqGADIezH7P7Qhy8OWMulo7X9LmkOhrEuOZYstnxxqPR00MoM4ptH8732I+8FpdTpP96D1N2QjtNFbKW7c= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778713519; c=relaxed/simple; bh=6QUt0tcgP+9oAsWmFpuTxXj3Qshfx2dAIQzY/1uoHgE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lxDMlx/QCv5xNVnvT3UVUQOIOxhAoIxQcehUBK/qWS+U47td2z20/g0pJl3/kaeOCuR5Udey9V8c5OvoVZSEc9CRT38Oe4QVcjuMtoybVJlOxqDgP4qI4EmJlyxTPre+tEXFUgWpwG0kw/i13WKdoRlZyu8WoRQCx+/7vwN+iKA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Drc1Y5Nh; arc=none smtp.client-ip=74.125.82.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Drc1Y5Nh" Received: by mail-dl1-f74.google.com with SMTP id a92af1059eb24-134cf343c58so643031c88.1 for ; Wed, 13 May 2026 16:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1778713517; x=1779318317; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TckraPBMpRT2yAffHT2vp2wfWUfkiwkTNolmaZdcwq8=; b=Drc1Y5NhiubSJRViPF1nsBXGlw9OTv1J5YP0xQOk9w7GmAgFAWrNrE+wCsYg+p2r4z r5l/rkoyNEdJgs3YzCeUY4dG3+isBjgs2MQWrIUP6EhiRX+330Eo1FONTN4MyJ6nEpO+ O+HP7LGY6ECDxB+20FkUcAjftUZnzPrOPwB/dfp6oAhMQKrNkVNYBq6uR5h+i3IcSvkg tF4ulBT16p89v8HEQQIYnIFbYBraH4spzdGfBvVzcL27uDrEAnZNPirFNpeUcGwzuh8i QTbkBqRzTKlSKQP7sST1E72OJQv1mTjPivo3ljunNnQOtP5EFeFYY2RgeGgbQufxqJsn WGnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778713517; x=1779318317; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TckraPBMpRT2yAffHT2vp2wfWUfkiwkTNolmaZdcwq8=; b=JbCF/a8F6piuXJo0e6sMJodDFewWc02BvIXvLVWw6HA3ri3Qqg763FVG+eN5wTpMk2 9WjjdYSw41Xh87CakAbTIyom5SEMLe2aL8cVSTWb27EWWmFOVpOVLxEK8M2v89S2lmA5 cBKSJ9LbbeFUh9enIcbbYv1E488z0xiti638FA9B8LZlmpiyPPn0T2rP82DHshyhOOOn w47TWyWh/mBHy6JfwrDzUMOstNjsAZIyxm/RBOe2GSDaeKTYBzy2QBvTTDJbPsJKBNLd 1lH9kBFfqetzLwhPpDs42pwurpZF2lunoDQoq1iE0jJWUttGf+Re4qouW9JxbehHBrsD 7Jkw== X-Forwarded-Encrypted: i=1; AFNElJ9wZGmuwURL6Ojllbr+dftWKoZokebEB3WWZacuN1TMqWamTV1tvwQG68kdB15YCObkQCDUkiJHENldzmy2Yrp5@vger.kernel.org X-Gm-Message-State: AOJu0Yxdpz844MWSydGpC0RsVlziIGULQSrQ7np7M4FogSiP1hBz7A1/ eZdsXJc8gXZN+NS1xFQLHYQtxabY96dnCLfvQX3H4X8rqoaQfdcKKYiEKHHCWcvEAjs7c4xCx0E SmkRx+hx6qw== X-Received: from dlad16.prod.google.com ([2002:a05:701b:2210:b0:130:c9dc:1b88]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7022:797:b0:132:5db9:27b3 with SMTP id a92af1059eb24-13436aa1684mr3349431c88.29.1778713516810; Wed, 13 May 2026 16:05:16 -0700 (PDT) Date: Wed, 13 May 2026 16:04:42 -0700 In-Reply-To: <20260513230450.529380-1-irogers@google.com> Precedence: bulk X-Mailing-List: linux-perf-users@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260513230450.529380-1-irogers@google.com> X-Mailer: git-send-email 2.54.0.563.g4f69b47b94-goog Message-ID: <20260513230450.529380-7-irogers@google.com> Subject: [PATCH v1 06/14] perf test: Refactor parallel poll loop to drain all pipes simultaneously From: Ian Rogers To: Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Alexander Shishkin , Jiri Olsa , Adrian Hunter , James Clark , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org Cc: Ian Rogers Content-Type: text/plain; charset="UTF-8" When running tests in parallel with verbose output (-v), child processes write to pipes. If a test produces significant output (e.g. Granite Rapids metric parsing printing hundreds of lines), it fills the 64KB pipe buffer and blocks. Previously, the parent harness (finish_test) only polled the pipe of the "current" test waiting to be printed. Other children blocked indefinitely until the parent reached them, severely sequentializing execution. Address this by implementing finish_tests_parallel() to poll and drain output pipes from all running children simultaneously into per-child buffers. Reaping occurs out of order as children finish, while final result printing remains strictly in order. This drops parallel verbose execution time for the PMU events suite from ~35 seconds down to ~5.9 seconds. Assisted-by: Gemini-CLI:Google Gemini 3 Signed-off-by: Ian Rogers --- tools/lib/subcmd/run-command.c | 4 +- tools/perf/tests/builtin-test.c | 189 +++++++++++++++++++++++++++++++- 2 files changed, 189 insertions(+), 4 deletions(-) diff --git a/tools/lib/subcmd/run-command.c b/tools/lib/subcmd/run-command.c index b7510f83209a..60e7df367316 100644 --- a/tools/lib/subcmd/run-command.c +++ b/tools/lib/subcmd/run-command.c @@ -241,8 +241,8 @@ int check_if_command_finished(struct child_process *cmd) sprintf(filename, "/proc/%u/status", cmd->pid); status_file = fopen(filename, "r"); if (status_file == NULL) { - /* Open failed assume finish_command was called. */ - return true; + /* Open failed. Only assume finished if process no longer exists. */ + return errno == ENOENT ? 1 : 0; } while (fgets(status_line, sizeof(status_line), status_file) != NULL) { char *p; diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c index a350db071674..ad1b941731ca 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -301,6 +301,9 @@ struct child_test { struct test_suite *test; int suite_num; int test_case_num; + struct strbuf err_output; + int result; + bool done; }; static jmp_buf run_test_jmp_buf; @@ -508,6 +511,187 @@ static void finish_test(struct child_test **child_tests, int running_test, int c zfree(&child_tests[running_test]); } +static int finish_tests_parallel(struct child_test **child_tests, size_t num_tests, int width) +{ + size_t next_to_print = 0; + struct pollfd *pfds; + size_t *pfd_indices; + size_t num_pfds = 0; +static void drain_child_process_err(struct child_test *child) +{ + char buf[512]; + ssize_t len; + + while ((len = read(child->process.err, buf, sizeof(buf) - 1)) > 0) { + buf[len] = '\0'; + strbuf_addstr(&child->err_output, buf); + } +} + +static int finish_tests_parallel(struct child_test **child_tests, size_t num_tests, int width) +{ + size_t next_to_print = 0; + struct pollfd *pfds; + size_t *pfd_indices; + size_t num_pfds = 0; + int last_running = -1; + size_t i; + int last_suite_printed = -1; + + pfds = calloc(num_tests, sizeof(*pfds)); + pfd_indices = calloc(num_tests, sizeof(*pfd_indices)); + if (!pfds || !pfd_indices) { + free(pfds); + free(pfd_indices); + return -ENOMEM; + } + + for (i = 0; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child) + continue; + strbuf_init(&child->err_output, 0); + if (child->process.err > 0) + fcntl(child->process.err, F_SETFL, O_NONBLOCK); + } + + while (next_to_print < num_tests) { + size_t running_count = 0; + size_t p; + + while (next_to_print < num_tests && + (!child_tests[next_to_print] || child_tests[next_to_print]->done)) + next_to_print++; + + if (next_to_print >= num_tests) + break; + + num_pfds = 0; + + for (i = next_to_print; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child || child->done) + continue; + + if (!check_if_command_finished(&child->process)) + running_count++; + + if (child->process.err > 0) { + pfds[num_pfds].fd = child->process.err; + pfds[num_pfds].events = POLLIN | POLLERR | POLLHUP | POLLNVAL; + pfd_indices[num_pfds] = i; + num_pfds++; + } + } + + if (perf_use_color_default && running_count != (size_t)last_running) { + struct child_test *next_child = child_tests[next_to_print]; + + if (last_running != -1) + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + + if (next_child) { + if (test_suite__num_test_cases(next_child->test) > 1 && + last_suite_printed != next_child->suite_num) { + pr_info("%3d: %-*s:\n", next_child->suite_num + 1, width, + test_description(next_child->test, -1)); + last_suite_printed = next_child->suite_num; + } + print_test_result(next_child->test, next_child->suite_num, + next_child->test_case_num, TEST_RUNNING, width, + running_count); + } + last_running = running_count; + } + + if (num_pfds == 0) { + if (running_count > 0) + usleep(10 * 1000); + } else { + int pret = poll(pfds, num_pfds, 100); + + if (pret > 0) { + for (p = 0; p < num_pfds; p++) { + if (pfds[p].revents) { + size_t idx = pfd_indices[p]; + struct child_test *child = child_tests[idx]; + + drain_child_process_err(child); + } + } + } + } + + for (i = next_to_print; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child || child->done) + continue; + + if (check_if_command_finished(&child->process)) { + if (child->process.err > 0) { + drain_child_process_err(child); + close(child->process.err); + child->process.err = -1; + } + child->result = finish_command(&child->process); + child->done = true; + } + } + + while (next_to_print < num_tests) { + struct child_test *child = child_tests[next_to_print]; + + if (!child) { + next_to_print++; + continue; + } + if (!child->done) + break; + + if (perf_use_color_default && last_running != -1) { + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + last_running = -1; + } + + if (test_suite__num_test_cases(child->test) > 1 && + last_suite_printed != child->suite_num) { + pr_info("%3d: %-*s:\n", child->suite_num + 1, width, + test_description(child->test, -1)); + last_suite_printed = child->suite_num; + } + + if (verbose > 1) { + if (test_suite__num_test_cases(child->test) > 1) { + pr_info("%3d.%1d: %s:\n", child->suite_num + 1, + child->test_case_num + 1, + test_description(child->test, + child->test_case_num)); + } else { + pr_info("%3d: %s:\n", child->suite_num + 1, + test_description(child->test, -1)); + } + } + + if (verbose > 1 || (verbose == 1 && child->result == TEST_FAIL)) + fprintf(stderr, "%s", child->err_output.buf); + + print_test_result(child->test, child->suite_num, child->test_case_num, + child->result, width, 0); + strbuf_release(&child->err_output); + child_tests[next_to_print] = NULL; + zfree(&child); + next_to_print++; + } + } + + free(pfds); + free(pfd_indices); + return 0; +} + static int start_test(struct test_suite *test, int curr_suite, int curr_test_case, struct child_test **child, int width, int pass) { @@ -670,8 +854,9 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], } if (!sequential) { /* Parallel mode starts tests but doesn't finish them. Do that now. */ - for (size_t x = 0; x < num_tests; x++) - finish_test(child_tests, x, num_tests, width); + err = finish_tests_parallel(child_tests, num_tests, width); + if (err) + goto err_out; } } err_out: -- 2.54.0.563.g4f69b47b94-goog