From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93758C433F5 for ; Thu, 9 Sep 2021 13:47:30 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 594A960041 for ; Thu, 9 Sep 2021 13:47:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 594A960041 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RdXWLzccbxFX1lvX5nTDJgQu0k2lbtJ+WkvG257RpZ4=; b=23E/Z99qBGR5Yg dtYQzxCJGs2+/h90SFFDeDwcax4uKJuycjgAoClVb1jjhZdIwcUNbepLpzfoUfB9yXglUfbzDd19+ Drbk2J4CaxuaDlZXTd/gLz4zHpl7FBJ7TiFHdqNZW+slmH4oqMt+R7I/BgWfO8vTQlSCwSb+8nFQJ iVC+qeDiT12hQfPMwSFobrdgFrOe1Ar20q+cTYI+mZtBE3mQcTOXNRL7HSXj1RgHMCJF0HiroUXEB G86xNX9Pc3551u+r4uIgr3DJNNs4I5HmuWtfOiN0vrkhKJ84mk7XhZUJ7Ih0wSa6QmrsYWqUMeO4p zV3XVxRWku8eRe0Na/FQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mOKMx-009zbv-HP; Thu, 09 Sep 2021 13:45:36 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mOKMp-009zZC-Dv for linux-arm-kernel@lists.infradead.org; Thu, 09 Sep 2021 13:45:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631195125; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xTpohogIA64F7YnEY632XtA1i/qGIAHCwBWGTfs60IE=; b=MnZZgNmFFDXmd4g+SuwJV/tgeJ+whW9f+zuAQcghz7LWRKW+KMEWK4nO0e42szmfodAZwZ Oo145YD+/OmgzU+vjo4svTmD3olNISlqit2u4MdqVFRvJ4qus+oxsNOH/Xagm5l1ZYCeCu pJDPXIclmhh/MZYEaHaT9LF+sFj4HjE= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-317-O-F2s_aVMBCuwNlcUQrNLA-1; Thu, 09 Sep 2021 09:45:24 -0400 X-MC-Unique: O-F2s_aVMBCuwNlcUQrNLA-1 Received: by mail-wm1-f70.google.com with SMTP id f17-20020a05600c155100b002f05f30ff03so896531wmg.3 for ; Thu, 09 Sep 2021 06:45:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=xTpohogIA64F7YnEY632XtA1i/qGIAHCwBWGTfs60IE=; b=gqL6FEAMdsl7Ib/N3DmGY27bJcBXa3rR3CdemkoqOc0AlIfbTifsycm9DOzj5TFVWn CMiyj4aAipUQ1XIBkJJCPX7pR8jn5WHvUPmPmHNYlrFtS5ekHEdwMRM/bLKkWN/TDevc rpx8gPoPQb0Uqc3POyf1p9Y/3gSlTGnmvd4TF7h1MUgKJigjhwSnqEqGdYQenrZIOxM3 W5XBqL5Gg9Hke3zLoZgMQrxibsAKuGn0agPZuQOc+k5sJa4Gq7ggsLGuG0Ynr78HF/9i RBOlDgwCn3JnAFF4cfbg/zvpk1qRJbe1v0weW1p+nFRnd2zGGtbIk38HHEgDyehuxqsx sWVg== X-Gm-Message-State: AOAM533LV+qBu9a7Ic1Ox07F8AWvJqBBbXaNx8hufsf5XUatRyrGcR9R osUaEA3DYI4n0VIer8i01LUlmwZ0kEaWhIHnZKmzHSFdXVpYH7/FtVmMmGB9IU2wTQs45uWjXww E0EynNNZIdXNzytKF6ax8D49N7HyNPNAmosI= X-Received: by 2002:a7b:cbc9:: with SMTP id n9mr3122667wmi.50.1631195123208; Thu, 09 Sep 2021 06:45:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx7puNMDXniSPYBQjdcRLIZDpZ0bmtEJYUCUsx+UIrFjPnaJ0zlntbh8UgDW66c97dJo2dZeQ== X-Received: by 2002:a7b:cbc9:: with SMTP id n9mr3122635wmi.50.1631195122967; Thu, 09 Sep 2021 06:45:22 -0700 (PDT) Received: from gator (cst2-174-132.cust.vodafone.cz. [31.30.174.132]) by smtp.gmail.com with ESMTPSA id u16sm1719047wmc.41.2021.09.09.06.45.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Sep 2021 06:45:22 -0700 (PDT) Date: Thu, 9 Sep 2021 15:45:20 +0200 From: Andrew Jones To: Raghavendra Rao Ananta Cc: Paolo Bonzini , Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Will Deacon , Peter Shier , Ricardo Koller , Oliver Upton , Reiji Watanabe , Jing Zhang , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Subject: Re: [PATCH v4 16/18] KVM: arm64: selftests: arch_timer: Support vCPU migration Message-ID: <20210909134520.yxrjestdwsishce2@gator> References: <20210909013818.1191270-1-rananta@google.com> <20210909013818.1191270-17-rananta@google.com> MIME-Version: 1.0 In-Reply-To: <20210909013818.1191270-17-rananta@google.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=drjones@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210909_064527_660159_C13E61E3 X-CRM114-Status: GOOD ( 37.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Sep 09, 2021 at 01:38:16AM +0000, Raghavendra Rao Ananta wrote: > Since the timer stack (hardware and KVM) is per-CPU, there > are potential chances for races to occur when the scheduler > decides to migrate a vCPU thread to a different physical CPU. > Hence, include an option to stress-test this part as well by > forcing the vCPUs to migrate across physical CPUs in the > system at a particular rate. > > Originally, the bug for the fix with commit 3134cc8beb69d0d > ("KVM: arm64: vgic: Resample HW pending state on deactivation") > was discovered using arch_timer test with vCPU migrations and > can be easily reproduced. > > Signed-off-by: Raghavendra Rao Ananta > --- > .../selftests/kvm/aarch64/arch_timer.c | 113 +++++++++++++++++- > 1 file changed, 112 insertions(+), 1 deletion(-) > > diff --git a/tools/testing/selftests/kvm/aarch64/arch_timer.c b/tools/testing/selftests/kvm/aarch64/arch_timer.c > index 6141c387e6dc..aac7bcea4352 100644 > --- a/tools/testing/selftests/kvm/aarch64/arch_timer.c > +++ b/tools/testing/selftests/kvm/aarch64/arch_timer.c > @@ -14,6 +14,8 @@ > * > * The test provides command-line options to configure the timer's > * period (-p), number of vCPUs (-n), and iterations per stage (-i). > + * To stress-test the timer stack even more, an option to migrate the > + * vCPUs across pCPUs (-m), at a particular rate, is also provided. > * > * Copyright (c) 2021, Google LLC. > */ > @@ -24,6 +26,8 @@ > #include > #include > #include > +#include > +#include > > #include "kvm_util.h" > #include "processor.h" > @@ -36,17 +40,20 @@ > #define NR_TEST_ITERS_DEF 5 > #define TIMER_TEST_PERIOD_MS_DEF 10 > #define TIMER_TEST_ERR_MARGIN_US 100 > +#define TIMER_TEST_MIGRATION_FREQ_MS 2 > > struct test_args { > int nr_vcpus; > int nr_iter; > int timer_period_ms; > + int migration_freq_ms; > }; > > static struct test_args test_args = { > .nr_vcpus = NR_VCPUS_DEF, > .nr_iter = NR_TEST_ITERS_DEF, > .timer_period_ms = TIMER_TEST_PERIOD_MS_DEF, > + .migration_freq_ms = TIMER_TEST_MIGRATION_FREQ_MS, > }; > > #define msecs_to_usecs(msec) ((msec) * 1000LL) > @@ -81,6 +88,9 @@ struct test_vcpu { > static struct test_vcpu test_vcpu[KVM_MAX_VCPUS]; > static struct test_vcpu_shared_data vcpu_shared_data[KVM_MAX_VCPUS]; > > +static unsigned long *vcpu_done_map; > +static pthread_mutex_t vcpu_done_map_lock; > + > static void > guest_configure_timer_action(struct test_vcpu_shared_data *shared_data) > { > @@ -216,6 +226,11 @@ static void *test_vcpu_run(void *arg) > > vcpu_run(vm, vcpuid); > > + /* Currently, any exit from guest is an indication of completion */ > + pthread_mutex_lock(&vcpu_done_map_lock); > + set_bit(vcpuid, vcpu_done_map); > + pthread_mutex_unlock(&vcpu_done_map_lock); > + > switch (get_ucall(vm, vcpuid, &uc)) { > case UCALL_SYNC: > case UCALL_DONE: > @@ -234,9 +249,76 @@ static void *test_vcpu_run(void *arg) > return NULL; > } > > +static uint32_t test_get_pcpu(void) > +{ > + uint32_t pcpu; > + unsigned int nproc_conf; > + cpu_set_t online_cpuset; > + > + nproc_conf = get_nprocs_conf(); > + sched_getaffinity(0, sizeof(cpu_set_t), &online_cpuset); > + > + /* Randomly find an available pCPU to place a vCPU on */ > + do { > + pcpu = rand() % nproc_conf; > + } while (!CPU_ISSET(pcpu, &online_cpuset)); > + > + return pcpu; > +} Missing blank line here. > +static int test_migrate_vcpu(struct test_vcpu *vcpu) > +{ > + int ret; > + cpu_set_t cpuset; > + uint32_t new_pcpu = test_get_pcpu(); > + > + CPU_ZERO(&cpuset); > + CPU_SET(new_pcpu, &cpuset); > + > + pr_debug("Migrating vCPU: %u to pCPU: %u\n", vcpu->vcpuid, new_pcpu); > + > + ret = pthread_setaffinity_np(vcpu->pt_vcpu_run, > + sizeof(cpuset), &cpuset); > + > + /* Allow the error where the vCPU thread is already finished */ > + TEST_ASSERT(ret == 0 || ret == ESRCH, > + "Failed to migrate the vCPU:%u to pCPU: %u; ret: %d\n", > + vcpu->vcpuid, new_pcpu, ret); > + > + return ret; > +} Missing blank line here. > +static void *test_vcpu_migration(void *arg) > +{ > + unsigned int i, n_done; > + bool vcpu_done; > + > + do { > + usleep(msecs_to_usecs(test_args.migration_freq_ms)); > + > + for (n_done = 0, i = 0; i < test_args.nr_vcpus; i++) { > + pthread_mutex_lock(&vcpu_done_map_lock); > + vcpu_done = test_bit(i, vcpu_done_map); > + pthread_mutex_unlock(&vcpu_done_map_lock); > + > + if (vcpu_done) { > + n_done++; > + continue; > + } > + > + test_migrate_vcpu(&test_vcpu[i]); > + } > + } while (test_args.nr_vcpus != n_done); > + > + return NULL; > +} > + > static void test_run(struct kvm_vm *vm) > { > int i, ret; > + pthread_t pt_vcpu_migration; > + > + pthread_mutex_init(&vcpu_done_map_lock, NULL); > + vcpu_done_map = bitmap_alloc(test_args.nr_vcpus); > + TEST_ASSERT(vcpu_done_map, "Failed to allocate vcpu done bitmap\n"); > > for (i = 0; i < test_args.nr_vcpus; i++) { > ret = pthread_create(&test_vcpu[i].pt_vcpu_run, NULL, > @@ -244,8 +326,23 @@ static void test_run(struct kvm_vm *vm) > TEST_ASSERT(!ret, "Failed to create vCPU-%d pthread\n", i); > } > > + /* Spawn a thread to control the vCPU migrations */ > + if (test_args.migration_freq_ms) { > + srand(time(NULL)); > + > + ret = pthread_create(&pt_vcpu_migration, NULL, > + test_vcpu_migration, NULL); > + TEST_ASSERT(!ret, "Failed to create the migration pthread\n"); > + } > + > + > for (i = 0; i < test_args.nr_vcpus; i++) > pthread_join(test_vcpu[i].pt_vcpu_run, NULL); > + > + if (test_args.migration_freq_ms) > + pthread_join(pt_vcpu_migration, NULL); > + > + bitmap_free(vcpu_done_map); > } > > static struct kvm_vm *test_vm_create(void) > @@ -286,6 +383,8 @@ static void test_print_help(char *name) > NR_TEST_ITERS_DEF); > pr_info("\t-p: Periodicity (in ms) of the guest timer (default: %u)\n", > TIMER_TEST_PERIOD_MS_DEF); > + pr_info("\t-m: Frequency (in ms) of vCPUs to migrate to different pCPU. 0 to turn off (default: %u)\n", > + TIMER_TEST_MIGRATION_FREQ_MS); > pr_info("\t-h: print this help screen\n"); > } > > @@ -293,7 +392,7 @@ static bool parse_args(int argc, char *argv[]) > { > int opt; > > - while ((opt = getopt(argc, argv, "hn:i:p:")) != -1) { > + while ((opt = getopt(argc, argv, "hn:i:p:m:")) != -1) { > switch (opt) { > case 'n': > test_args.nr_vcpus = atoi(optarg); > @@ -320,6 +419,13 @@ static bool parse_args(int argc, char *argv[]) > goto err; > } > break; > + case 'm': > + test_args.migration_freq_ms = atoi(optarg); > + if (test_args.migration_freq_ms < 0) { > + pr_info("0 or positive value needed for -m\n"); > + goto err; > + } > + break; > case 'h': > default: > goto err; > @@ -343,6 +449,11 @@ int main(int argc, char *argv[]) > if (!parse_args(argc, argv)) > exit(KSFT_SKIP); > > + if (get_nprocs() < 2) { Even though the chance of being on a uniprocessor is low and the migration test is now on by default, we could still do if (test_args.migration_freq_ms && get_nprocs() < 2) since that's why the skip message says it needs two cpus. > + print_skip("At least two physical CPUs needed for vCPU migration"); > + exit(KSFT_SKIP); > + } > + > vm = test_vm_create(); > test_run(vm); > kvm_vm_free(vm); > -- > 2.33.0.153.gba50c8fa24-goog > Reviewed-by: Andrew Jones Thanks, drew _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel