From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E34D22C6B7 for ; Fri, 12 Jul 2024 20:02:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720814576; cv=none; b=Wbm/VQRdzwto2Fz9ceRY9MthUl+92+21gUx9fmmnHgu2GZ+J6/tIAX5ccl2vWb9A72uLE5Cs4OdfMlDSfjzy+WA9rVb1w2qnWBKSIdOqZDfkpHx7GrFfMQhcHYU0CgH4s3EpCzNtrUxs1F76ELI6CRSH4ZTGlS49fWOwdUTUhNY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720814576; c=relaxed/simple; bh=1zUPhysoViSxlYcRngclA3bOaQJucOi1Pj/v9a+PX74=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type:Content-Disposition; b=MkvL3Rxts+pdDclXBScqtZELtAryF/JZIFaIxYUokn6z8GLCkTkDHEv787gt1Vl07CjHV8PVCgl3kDtS65pZ3gy1ECxSUFC/XKCdNIxkgRQukHc+bbmRxfb1pDHHt0Dz+V81QxCCxE+rGUpq7jeZDjbpVDuQBggCTmEVkG1Jyc4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ZGDt0YzQ; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZGDt0YzQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720814573; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WALHv74WV2cVjzZCL39eKCy6GimLkLb+2S99yNQNkn0=; b=ZGDt0YzQ7yBDe6FQzeSmt5qDS6ByfGKhz9KbZY91uby0W2w63mksgM00Msa4VWWW8df+N9 csWWvZ/UwA3c8ZCw737HsFCAqq03mjQHqqCEAiMZOeQj1JFyC9HicjN2zKWdHNTfw+nAdI KCGZFVS7Q3HABV+6q7tgg+RChkx3H3Q= Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-37-dRU8d-8vPLmYRmMmrN_f4w-1; Fri, 12 Jul 2024 16:02:52 -0400 X-MC-Unique: dRU8d-8vPLmYRmMmrN_f4w-1 Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-70b2793d2ffso2206963b3a.2 for ; Fri, 12 Jul 2024 13:02:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720814571; x=1721419371; h=content-transfer-encoding:content-disposition:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WALHv74WV2cVjzZCL39eKCy6GimLkLb+2S99yNQNkn0=; b=k9kndivRH8jT8mlGebATnf4WShkI9YMBmmfzoWPLMsZaXVIp77Suisjs4a9J/d9csD mo5XIB7tCr21M8k2duvUFtEFuNoxnTFm3SvCUThean9tFfpDG36Zh5OOmtqPz3MeUlU+ BhLOEd9htMHKrs38g8HsV/Ml7y8HMQ/o4VeUfsORhwhYIVcX+PUu0U8T2ROKp0Z3nPXC bPo3ZZvC7ps53/t9IcHqG3GX1VzlX7JGtC4dXteW0Rn3zjALinwHpA0IC8WsrRQptgN3 BRUNtIyina0o9Q8LUKsJ3iEh1p6Q3jOf6wYllf0JtgQXcMs05NLELoE+XQpLRHWvJt2g t4Uw== X-Forwarded-Encrypted: i=1; AJvYcCXPPN1QGWujDG2W2hx/rp5FwWJhI3ymcFbfQRC+sO6+a494VJL5NYZLsq7mZ4fzH8aYKMu8wrVhnzB5I3cbg7e4i3E11V6/WkVIuWw/ X-Gm-Message-State: AOJu0YzeRNEc3c2+8DJ66ADP96IVgqDxmlNOOFL6aP7QExuSy+c8dvvQ QGBjZmFiEntW4yNpbxFjZw82+0CBXOyzqk+qjcj83MLmQVe2OXccIZDuK+tOK8UnjpM9l9flZdg hJ3oxRkHZGbXI4XOAs18d2FzyZtE3BE6D9NKU88+NB1O3k4smzlWgKhVBoD81rQ== X-Received: by 2002:a05:6a00:3e25:b0:705:9fc7:9133 with SMTP id d2e1a72fcca58-70b435e7ee2mr15391780b3a.23.1720814571359; Fri, 12 Jul 2024 13:02:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFNmlk9BsIdDCqjINYXL5lLBoE++woDa8pnC7jaIuaU38uh72LjbkqaARCHNWnS5/S8BJKC3A== X-Received: by 2002:a05:6a00:3e25:b0:705:9fc7:9133 with SMTP id d2e1a72fcca58-70b435e7ee2mr15391739b3a.23.1720814570900; Fri, 12 Jul 2024 13:02:50 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a802:5d71:55fb:289:f049:5d12]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70b43967791sm7901658b3a.118.2024.07.12.13.02.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Jul 2024 13:02:50 -0700 (PDT) From: Leonardo Bras To: Paolo Bonzini Cc: Leonardo Bras , "Paul E. McKenney" , Leonardo Bras , Sean Christopherson , Frederic Weisbecker , Marcelo Tosatti , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Subject: Re: [RFC PATCH 1/1] kvm: Note an RCU quiescent state on guest exit Date: Fri, 12 Jul 2024 17:02:30 -0300 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: <68c39823-6b1d-4368-bd1e-a521ade8889b@paulmck-laptop> <17ebd54d-a058-4bc8-bd65-a175d73b6d1a@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: 8bit On Fri, Jul 12, 2024 at 05:57:10PM +0200, Paolo Bonzini wrote: > On 7/11/24 01:18, Leonardo Bras wrote: > > What are your thoughts on above results? > > Anything you would suggest changing? > Hello Paolo, thanks for the feedback! > Can you run the test with a conditional on "!tick_nohz_full_cpu(vcpu->cpu)"? > > If your hunch is correct that nohz-full CPUs already avoid invoke_rcu_core() > you might get the best of both worlds. > > tick_nohz_full_cpu() is very fast when there is no nohz-full CPU, because > then it shortcuts on context_tracking_enabled() (which is just a static > key). But that would mean not noting an RCU quiescent state in guest_exit of nohz_full cpus, right? The original issue we were dealing was having invoke_rcu_core() running on nohz_full cpus, and messing up the latency of RT workloads inside the VM. While most of the invoke_rcu_core() get ignored by the nohz_full rule, there are some scenarios in which it the vcpu thread may take more than 1s between a guest_entry and the next one (VM busy), and those which did not get ignored have caused latency peaks in our tests. The main idea of this patch is to note RCU quiescent states on guest_exit at nohz_full cpus (and use rcu.patience) to avoid running invoke_rcu_core() between a guest_exit and the next guest_entry if it takes less than rcu.patience miliseconds between exit and entry, and thus avoiding the latency increase. What I tried to prove above is that it also improves non-Isolated cores as well, since rcu_core will not be running as often, saving cpu cycles that can be used by the VM. What are your thoughts on that? Thanks! Leo