From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE37AC27C76 for ; Fri, 20 Jan 2023 22:12:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229949AbjATWMh (ORCPT ); Fri, 20 Jan 2023 17:12:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjATWMg (ORCPT ); Fri, 20 Jan 2023 17:12:36 -0500 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75BF036449 for ; Fri, 20 Jan 2023 14:12:34 -0800 (PST) Received: by mail-io1-xd2d.google.com with SMTP id y69so3110852iof.3 for ; Fri, 20 Jan 2023 14:12:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=digitalocean.com; s=google; h=cc:to:message-id:date:content-transfer-encoding:mime-version :subject:from:from:to:cc:subject:date:message-id:reply-to; bh=U1Ao9NXaWgBOUiK+rj9v9X8Q1xZNi+V9uTD46WKKtdw=; b=CP69oNJLnjiVV4tB0rYNSmBoYhq8zTAf5aeWGwBovXoy78p1YHNttTfFLpgUdkZt5y CF6BkjRK5BXmetvzcOipE6ZWALHWD4fDKhxrzQDRZETLKzd8lYiiMlj8HPAiBZDkBq+1 Fjdww4iDsWbfzKLRwjC85+Yo6BGDTWL5y5Tv4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:message-id:date:content-transfer-encoding:mime-version :subject:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=U1Ao9NXaWgBOUiK+rj9v9X8Q1xZNi+V9uTD46WKKtdw=; b=16AXSckU+8K5kbk/Spw/MvDHGRGeYzOIRy2uiRV6Mgmt7A0Lgu52dqYT3zumQ1TPIr rfaC79zqij/SWRNBBNMpuO13wwjEto1BVw23aGaeAWmjqbV5Q2IRA0wFHJ2duy6ku7fL /Dx4sWzF3hyrAz88hNMlgmgodoz9HCC+jcUmQRqGPKnbwJwbdCoXsidagFgk19QDIj5h Di6SowknVach2k8TSJ2IbEZv8bXyMe4wlEyrecfCmXtoesOdqkFCdyuWJkTl4TbzFtJH miNJHX2qDfYpWd0vlN6TP9dFQtTrvpezzex+dGjJWT/Nx3lIpYhVXuswAx9vvVTlR0Ch tgjg== X-Gm-Message-State: AFqh2krljOwOUsMEDNg9WEfMjiwT2l7J1xRpOF/pMqOHbtD4pw/RPoJ4 Yf/RhMfO2KBVl0XtC4sGyAdBig== X-Google-Smtp-Source: AMrXdXum9CTPzdCUTloBWrH1lZInZCJwTTZUYqpFqr+e9Y/2n3mTCcLg6hp+gsZAesCShxp7/GqsWA== X-Received: by 2002:a6b:500a:0:b0:705:5e1e:eb6e with SMTP id e10-20020a6b500a000000b007055e1eeb6emr9205488iob.11.1674252753715; Fri, 20 Jan 2023 14:12:33 -0800 (PST) Received: from localhost ([136.37.131.79]) by smtp.gmail.com with ESMTPSA id cs8-20020a056638470800b003a7cadffda7sm1519487jab.2.2023.01.20.14.12.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 14:12:33 -0800 (PST) From: "Seth Forshee (DigitalOcean)" X-Google-Original-From: "Seth Forshee (DigitalOcean)" Subject: [PATCH 0/2] vhost: improve livepatch switching for heavily loaded vhost worker kthreads MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-b4-tracking: H4sIAMQRy2MC/w3LOw6AIAwA0KuYzjZBWNTbFGykkVRD/QzGu8v4hveCcRU2mLsXKt9ismvD0HeQMu nKKEszeOeDG7zDO+924lYOtEfOlEVXjDRRIA5jHAnajGSMsZKm3K5epXzfD5ena69qAAAA Date: Fri, 20 Jan 2023 16:12:20 -0600 Message-Id: <20230120-vhost-klp-switching-v1-0-7c2b65519c43@kernel.org> To: Petr Mladek , Jason Wang , "Michael S. Tsirkin" , Jiri Kosina , Miroslav Benes , Joe Lawrence , Josh Poimboeuf Cc: virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, "Seth Forshee (DigitalOcean)" , netdev@vger.kernel.org, live-patching@vger.kernel.org, linux-kernel@vger.kernel.org X-Mailer: b4 0.10.1 X-Developer-Signature: v=1; a=openpgp-sha256; l=2416; i=sforshee@kernel.org; h=from:subject:message-id; bh=EZbjAIdznm3Oej+eWIGyRgmoqSdbR0GomfdbWyUvWuE=; b=owEBbQGS/pANAwAKAVMDma7l9DHJAcsmYgBjyxHK47dK5XiK+U7PMs1DUY+EH512EmtrWsinE62O st1NWfiJATMEAAEKAB0WIQSQnt+rKAvnETy4Hc9TA5mu5fQxyQUCY8sRygAKCRBTA5mu5fQxySm3B/ 9RzhyLuYXteP+GKtAYPSH91mkV9to22qctt0HVI4O7jF/xfnsSaW4H0H02HzXQL4C8vk9TICOgCpWM pXpuuKMmpauwn5I88hvtmnChrAtfuwXV9F/UZOx/bcGLSB0XyBo6ZeZhGFtiBSptCzz8vAjY181OnP DAN50d3RjfNYScU+jrA3MfX9R0/PfKs71zuNpV02R961kdFFUwIS/tPPHJHkElpqV/jm9BLIkCYle/ Bwg0bcI2xNiyjrSUqq5rr8jayRwWFBpLFlz3HgMSaIugOyfVtjr2ZTXwNjwmvhBPu7+KsrMWT/MUWO 5AS2iykCZxhYrlFLHs79+KNjXkreR5 X-Developer-Key: i=sforshee@kernel.org; a=openpgp; fpr=2ABCA7498D83E1D32D51D3B5AB4800A62DB9F73A Precedence: bulk List-ID: X-Mailing-List: live-patching@vger.kernel.org We've fairly regularaly seen liveptches which cannot transition within kpatch's timeout period due to busy vhost worker kthreads. In looking for a solution the only answer I found was to call klp_update_patch_state() from a safe location. I tried adding this call to vhost_worker(), and it works, but this creates the potential for problems if a livepatch attempted to patch vhost_worker(). Without a call to klp_update_patch_state() fully loaded vhost kthreads can never switch because vhost_worker() will always appear on the stack, but with the call these kthreads can switch but will still be running the old version of vhost_worker(). To avoid this situation I've added a new function, klp_switch_current(), which switches the current task only if its stack does not include any function being patched. This allows kthreads to safely attempt switching themselves if a patch is pending. There is at least one downside, however. Since there's no way for the kthread to track whether it has already tried to switch for a pending patch it can end up calling klp_switch_current() repeatedly when it can never be safely switched. I don't know whether this is the right solution, and I'm happy to try out other suggestions. But in my testing these patches proved effective in consistently switching heavily loaded vhost kthreads almost immediately. To: Josh Poimboeuf To: Jiri Kosina To: Miroslav Benes To: Petr Mladek To: Joe Lawrence To: "Michael S. Tsirkin" To: Jason Wang Cc: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: netdev@vger.kernel.org Signed-off-by: Seth Forshee (DigitalOcean) --- Seth Forshee (DigitalOcean) (2): livepatch: add an interface for safely switching kthreads vhost: check for pending livepatches from vhost worker kthreads drivers/vhost/vhost.c | 4 ++++ include/linux/livepatch.h | 2 ++ kernel/livepatch/transition.c | 11 +++++++++++ 3 files changed, 17 insertions(+) --- base-commit: 5dc4c995db9eb45f6373a956eb1f69460e69e6d4 change-id: 20230120-vhost-klp-switching-ba9a3ae38b8a Best regards, -- Seth Forshee (DigitalOcean)