From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Fri, 1 Dec 2023 16:51:55 -0800 Subject: [PATCH 05/26] vfio: KVM: Pass get/put helpers from KVM to VFIO, don't do circular lookup In-Reply-To: <20230918160258.GL13795@ziepe.ca> References: <20230916003118.2540661-1-seanjc@google.com> <20230916003118.2540661-6-seanjc@google.com> <20230918152110.GI13795@ziepe.ca> <20230918160258.GL13795@ziepe.ca> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > On Mon, Sep 18, 2023 at 08:49:57AM -0700, Sean Christopherson wrote: > > On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > > > On Fri, Sep 15, 2023 at 05:30:57PM -0700, Sean Christopherson wrote: > > > > Explicitly pass KVM's get/put helpers to VFIO when attaching a VM to > > > > VFIO instead of having VFIO do a symbol lookup back into KVM. Having both > > > > KVM and VFIO do symbol lookups increases the overall complexity and places > > > > an unnecessary dependency on KVM (from VFIO) without adding any value. > > > > > > > > Signed-off-by: Sean Christopherson > > > > --- > > > > drivers/vfio/vfio.h | 2 ++ > > > > drivers/vfio/vfio_main.c | 74 +++++++++++++++++++--------------------- > > > > include/linux/vfio.h | 4 ++- > > > > virt/kvm/vfio.c | 9 +++-- > > > > 4 files changed, 47 insertions(+), 42 deletions(-) > > > > > > I don't mind this, but Christoph had disliked my prior attempt to do > > > this with function pointers.. > > > > > > The get can be inlined, IIRC, what about putting a pointer to the put > > > inside the kvm struct? > > > > That wouldn't allow us to achieve our goal, which is to hide the details of > > "struct kvm" from VFIO (and the rest of the kernel). > > > What's the objection to handing VFIO a function pointer? > > Hmm, looks like it was this thread: > > https://lore.kernel.org/r/0-v1-33906a626da1+16b0-vfio_kvm_no_group_jgg at nvidia.com > > Your rational looks a little better to me. > > > > The the normal kvm get/put don't have to exported symbols at all? > > > > The export of kvm_get_kvm_safe() can go away (I forgot to do that in this series), > > but kvm_get_kvm() will hang around as it's needed by KVM sub-modules (PPC and x86), > > KVMGT (x86), and drivers/s390/crypto/vfio_ap_ops.c (no idea what to call that beast). > > My thought would be to keep it as an inline, there should be some way > to do that without breaking your desire to hide the bulk of the kvm > struct content. Like put the refcount as the first element in the > struct and just don't ifdef it away?. That doesn't work because of the need to invoke kvm_destroy_vm() when the last reference is put, i.e. all of kvm_destroy_vm() would need to be inlined (LOL) or VFIO would need to do a symbol lookup on kvm_destroy_vm(), which puts back us at square one. There's one more wrinkle: this patch is buggy in that it doesn't ensure the liveliness of KVM-the-module, i.e. nothing prevents userspace from unloading kvm.ko while VFIO still holds a reference to a kvm structure, and so invoking ->put_kvm() could jump into freed code. To fix that, KVM would also need to pass along a module pointer :-( One thought would be to have vac.ko (tentative name), which is the "base" module that will hold the KVM/virtualization bits that need to be singletons, i.e. can't be per-KVM, provide the symbols needed for VFIO to manage references. But that just ends up moving the module reference trickiness into VAC+KVM, e.g. vac.ko would still need to be handed a function pointer in order to call back into the correct kvm.ko code. Hrm, but I suspect the vac.ko <=> kvm.ko interactions will need to deal with module shenanigans anyways, and the shenanigans would only affect crazy people like us that actually want multiple KVM modules. I'll plan on going that route. The very worst case scenario is that it just punts this conversation down to a possibile future. Dropping this patch and the previous prep patch won't meaningful affect the goals of this series, as only kvm_get_kvm_safe() and kvm_get_kvm() would need to be exposed outside of #ifdef __KVM__. Then we can figure out what to do with them if/when the whole multi-KVM thing comes along. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2132C7E2 for ; Sat, 2 Dec 2023 00:51:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yRU2yiPf" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-6cdedef4b62so3080954b3a.0 for ; Fri, 01 Dec 2023 16:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701478317; x=1702083117; darn=lists.linux.dev; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=yRU2yiPfeSTJGTLu4S/AB7ySnxlUSzTeZG6MJYpmPYeJz+KKnUh9uCoWQWcnammEyr JB8iXkJDjynNFqpcLmspSbeea2TG+lbygiDGjuKwz7xZ8naDI687uR0fgXHWQES6F565 3Wj5peNpYUeBAt1DJ6lixSaIFh8kyrIQY0zQcnGCF4LkOKHZtRKvnUbE9CeYmZixnml+ m9CBivcPEudrgIjSvtRKiD7G2IeSeKW2hLmL8KkCOH7rmC+bW4uVj8yRR6YrlpLpsZen lkCrnMpxkhUn314UNoLfbMC61HgEA4OGkAAn0P8qNZsk7hHuZL5HuBcI4mzRgB9RVchh KAxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701478317; x=1702083117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=v/K1Am2E/4SigQIsaI2feo7V1OKcMRY8K4oSc5TyaJgK2okWrSijJz7r3ekL2FUFbK TFnH3gCT9MsfEWi67pIB+5JMgS/d1ceyVi7iEI82T/FPAST3IPI6k7w2Yeu1lABZK3VV oqVU3f0HJMuoFRF5hQvrPXl1rqw6aubIXjs7nsaEpPQKwOgSyqfjCQFXvKqXn6GLk0AX 4Co9xN7g17yP9kJgp8/7jeDKc3BueNpSHUmFpv3XXwRvYahSjAUj+2x9gSM8UNHjpo+B ZNM3CNeAj0hLdJjNzKC1opseqFgLiKfTL1hlUqOlyx9ftoW5ta5DRzq2ERgc17Orf4bc je7Q== X-Gm-Message-State: AOJu0YwcAT7VwgrAQIHBBHOdSE9kckuo0qH9c3pj1tsa54fbkNWft5Lo JtmMOdka0aYSTE4bHfpFrcDweoyViQE= X-Google-Smtp-Source: AGHT+IEloABSTDcKm+41L5WpzazJRfhcCVpuae9X+15kKFthyi+8YQcZOmegDR+VoF+o36yu4wG4kUpnsoo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:5848:0:b0:5b8:fe99:152d with SMTP id i8-20020a635848000000b005b8fe99152dmr4028549pgm.7.1701478317322; Fri, 01 Dec 2023 16:51:57 -0800 (PST) Date: Fri, 1 Dec 2023 16:51:55 -0800 In-Reply-To: <20230918160258.GL13795@ziepe.ca> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230916003118.2540661-1-seanjc@google.com> <20230916003118.2540661-6-seanjc@google.com> <20230918152110.GI13795@ziepe.ca> <20230918160258.GL13795@ziepe.ca> Message-ID: Subject: Re: [PATCH 05/26] vfio: KVM: Pass get/put helpers from KVM to VFIO, don't do circular lookup From: Sean Christopherson To: Jason Gunthorpe Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Peter Zijlstra , Arnaldo Carvalho de Melo , Paolo Bonzini , Tony Krowiak , Halil Pasic , Jason Herne , Harald Freudenberger , Alex Williamson , Andy Lutomirski , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Anish Ghulati , Venkatesh Srinivas , Andrew Thornton Content-Type: text/plain; charset="us-ascii" On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > On Mon, Sep 18, 2023 at 08:49:57AM -0700, Sean Christopherson wrote: > > On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > > > On Fri, Sep 15, 2023 at 05:30:57PM -0700, Sean Christopherson wrote: > > > > Explicitly pass KVM's get/put helpers to VFIO when attaching a VM to > > > > VFIO instead of having VFIO do a symbol lookup back into KVM. Having both > > > > KVM and VFIO do symbol lookups increases the overall complexity and places > > > > an unnecessary dependency on KVM (from VFIO) without adding any value. > > > > > > > > Signed-off-by: Sean Christopherson > > > > --- > > > > drivers/vfio/vfio.h | 2 ++ > > > > drivers/vfio/vfio_main.c | 74 +++++++++++++++++++--------------------- > > > > include/linux/vfio.h | 4 ++- > > > > virt/kvm/vfio.c | 9 +++-- > > > > 4 files changed, 47 insertions(+), 42 deletions(-) > > > > > > I don't mind this, but Christoph had disliked my prior attempt to do > > > this with function pointers.. > > > > > > The get can be inlined, IIRC, what about putting a pointer to the put > > > inside the kvm struct? > > > > That wouldn't allow us to achieve our goal, which is to hide the details of > > "struct kvm" from VFIO (and the rest of the kernel). > > > What's the objection to handing VFIO a function pointer? > > Hmm, looks like it was this thread: > > https://lore.kernel.org/r/0-v1-33906a626da1+16b0-vfio_kvm_no_group_jgg@nvidia.com > > Your rational looks a little better to me. > > > > The the normal kvm get/put don't have to exported symbols at all? > > > > The export of kvm_get_kvm_safe() can go away (I forgot to do that in this series), > > but kvm_get_kvm() will hang around as it's needed by KVM sub-modules (PPC and x86), > > KVMGT (x86), and drivers/s390/crypto/vfio_ap_ops.c (no idea what to call that beast). > > My thought would be to keep it as an inline, there should be some way > to do that without breaking your desire to hide the bulk of the kvm > struct content. Like put the refcount as the first element in the > struct and just don't ifdef it away?. That doesn't work because of the need to invoke kvm_destroy_vm() when the last reference is put, i.e. all of kvm_destroy_vm() would need to be inlined (LOL) or VFIO would need to do a symbol lookup on kvm_destroy_vm(), which puts back us at square one. There's one more wrinkle: this patch is buggy in that it doesn't ensure the liveliness of KVM-the-module, i.e. nothing prevents userspace from unloading kvm.ko while VFIO still holds a reference to a kvm structure, and so invoking ->put_kvm() could jump into freed code. To fix that, KVM would also need to pass along a module pointer :-( One thought would be to have vac.ko (tentative name), which is the "base" module that will hold the KVM/virtualization bits that need to be singletons, i.e. can't be per-KVM, provide the symbols needed for VFIO to manage references. But that just ends up moving the module reference trickiness into VAC+KVM, e.g. vac.ko would still need to be handed a function pointer in order to call back into the correct kvm.ko code. Hrm, but I suspect the vac.ko <=> kvm.ko interactions will need to deal with module shenanigans anyways, and the shenanigans would only affect crazy people like us that actually want multiple KVM modules. I'll plan on going that route. The very worst case scenario is that it just punts this conversation down to a possibile future. Dropping this patch and the previous prep patch won't meaningful affect the goals of this series, as only kvm_get_kvm_safe() and kvm_get_kvm() would need to be exposed outside of #ifdef __KVM__. Then we can figure out what to do with them if/when the whole multi-KVM thing comes along. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62B5EC4167B for ; Sat, 2 Dec 2023 00:52:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=AmVhzq+Nbafh1AsxCAr7SFmSndvH11sBVcidKLrXMAo=; b=izzSTQNxynObZIOOtfRFQiuq3M eClFXsuVwMJfSiz56VBRZKtuw5NyvZpwpefcGYsNOEhbIokt/AKxR0gYaIKIuUAbGy7B/yfjmf1WF JJ6ICl2UmcpsxIyDtIhsNo+mx0NPfre2u44tZa1csnteg49IOcHqvtwY3Ik4gEGngMxWQx8ywGQaC tG9uNIP1tM/Yghq+RCJBsWVhGwRPqzxTIi3nDVYmPBFVe4GfFABq8COf0I7eodQUsex6DaOA06EHD zSfSL1T6TnEpAht3fGn5hxrvVPCPJVeLzHpwDhS1jaMYEdyjNDkFtYkS+X0uBjhrorm9jIVAjKnkd QGmd1Xuw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r9EEr-00Ewed-1j; Sat, 02 Dec 2023 00:52:09 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r9EEo-00EwcK-1A for linux-riscv@lists.infradead.org; Sat, 02 Dec 2023 00:52:08 +0000 Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-6cdedef4b62so3080956b3a.0 for ; Fri, 01 Dec 2023 16:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701478317; x=1702083117; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=ssmpraemfge8cj9yx0bQe0olitdcprZSjL66pJDSdmW0aPZIfyKM0ZJp9V1tq2m8jj J1Ua1wVh0BVpFthTM9gr9XJBNyjlSD9Ay0NcihLc7oazbRDj84VHrGecjs/ZuYvbyAnM Irwag3DSPVoTeWXxyK+mpZdjdQUXh6FtVOCjwaRgdE8+X6ITCVI3DOrACYY4mY9DvgHd YzTzmsNR8OIBrxP2rimNl3luKd5BOiGlJCgH+urgFV4uBABJzgaRJHiXbCpaiuG6xbQ7 BywurV+G1BbrK6d3YrA1uUATEdddTUsB4QDF7q6w8TBFaPUbMiOh9cC0cliWt+7ajyYC kUQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701478317; x=1702083117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=eNocGrtQvkaiRaIc8ijbY30rIPluYBekG/rN0nVSgj56NdaOHgIc7ZmebG+i45rsgF zpwY7IAktXXZBFou5rCKZcgGiSaWIG8c1G7Q/XNihyZv6pHkkGD/dsmtQ/84G122bMjw r5UoJdLy3GvxDANOviTHy//dPLRFjob1uayJpLx6RwCQWqZE9pBVUNr2xYxJLMpw0z1+ YrUebx70o8rW+gve99S74txNMjz8nkV7gc/l3CCTkjMPmiNeO8dpaAQIylLIRhrLOwHG KPlfsLiZF1CyVedNhsKPy2+6ZMCsvObA4WeyJMGMKLHROJq4o4YPYksJoODy8pGVMGv1 K/SA== X-Gm-Message-State: AOJu0Yz5kTMQ9JYOTU4FfotnWgYPuZfv1mLDcL1stIJU2NFl3G+Sf/l9 roPaH7TJl9e8elN1m1ialv0dbkkWr0g= X-Google-Smtp-Source: AGHT+IEloABSTDcKm+41L5WpzazJRfhcCVpuae9X+15kKFthyi+8YQcZOmegDR+VoF+o36yu4wG4kUpnsoo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:5848:0:b0:5b8:fe99:152d with SMTP id i8-20020a635848000000b005b8fe99152dmr4028549pgm.7.1701478317322; Fri, 01 Dec 2023 16:51:57 -0800 (PST) Date: Fri, 1 Dec 2023 16:51:55 -0800 In-Reply-To: <20230918160258.GL13795@ziepe.ca> Mime-Version: 1.0 References: <20230916003118.2540661-1-seanjc@google.com> <20230916003118.2540661-6-seanjc@google.com> <20230918152110.GI13795@ziepe.ca> <20230918160258.GL13795@ziepe.ca> Message-ID: Subject: Re: [PATCH 05/26] vfio: KVM: Pass get/put helpers from KVM to VFIO, don't do circular lookup From: Sean Christopherson To: Jason Gunthorpe Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Peter Zijlstra , Arnaldo Carvalho de Melo , Paolo Bonzini , Tony Krowiak , Halil Pasic , Jason Herne , Harald Freudenberger , Alex Williamson , Andy Lutomirski , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Anish Ghulati , Venkatesh Srinivas , Andrew Thornton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231201_165206_397128_68121A62 X-CRM114-Status: GOOD ( 34.70 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > On Mon, Sep 18, 2023 at 08:49:57AM -0700, Sean Christopherson wrote: > > On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > > > On Fri, Sep 15, 2023 at 05:30:57PM -0700, Sean Christopherson wrote: > > > > Explicitly pass KVM's get/put helpers to VFIO when attaching a VM to > > > > VFIO instead of having VFIO do a symbol lookup back into KVM. Having both > > > > KVM and VFIO do symbol lookups increases the overall complexity and places > > > > an unnecessary dependency on KVM (from VFIO) without adding any value. > > > > > > > > Signed-off-by: Sean Christopherson > > > > --- > > > > drivers/vfio/vfio.h | 2 ++ > > > > drivers/vfio/vfio_main.c | 74 +++++++++++++++++++--------------------- > > > > include/linux/vfio.h | 4 ++- > > > > virt/kvm/vfio.c | 9 +++-- > > > > 4 files changed, 47 insertions(+), 42 deletions(-) > > > > > > I don't mind this, but Christoph had disliked my prior attempt to do > > > this with function pointers.. > > > > > > The get can be inlined, IIRC, what about putting a pointer to the put > > > inside the kvm struct? > > > > That wouldn't allow us to achieve our goal, which is to hide the details of > > "struct kvm" from VFIO (and the rest of the kernel). > > > What's the objection to handing VFIO a function pointer? > > Hmm, looks like it was this thread: > > https://lore.kernel.org/r/0-v1-33906a626da1+16b0-vfio_kvm_no_group_jgg@nvidia.com > > Your rational looks a little better to me. > > > > The the normal kvm get/put don't have to exported symbols at all? > > > > The export of kvm_get_kvm_safe() can go away (I forgot to do that in this series), > > but kvm_get_kvm() will hang around as it's needed by KVM sub-modules (PPC and x86), > > KVMGT (x86), and drivers/s390/crypto/vfio_ap_ops.c (no idea what to call that beast). > > My thought would be to keep it as an inline, there should be some way > to do that without breaking your desire to hide the bulk of the kvm > struct content. Like put the refcount as the first element in the > struct and just don't ifdef it away?. That doesn't work because of the need to invoke kvm_destroy_vm() when the last reference is put, i.e. all of kvm_destroy_vm() would need to be inlined (LOL) or VFIO would need to do a symbol lookup on kvm_destroy_vm(), which puts back us at square one. There's one more wrinkle: this patch is buggy in that it doesn't ensure the liveliness of KVM-the-module, i.e. nothing prevents userspace from unloading kvm.ko while VFIO still holds a reference to a kvm structure, and so invoking ->put_kvm() could jump into freed code. To fix that, KVM would also need to pass along a module pointer :-( One thought would be to have vac.ko (tentative name), which is the "base" module that will hold the KVM/virtualization bits that need to be singletons, i.e. can't be per-KVM, provide the symbols needed for VFIO to manage references. But that just ends up moving the module reference trickiness into VAC+KVM, e.g. vac.ko would still need to be handed a function pointer in order to call back into the correct kvm.ko code. Hrm, but I suspect the vac.ko <=> kvm.ko interactions will need to deal with module shenanigans anyways, and the shenanigans would only affect crazy people like us that actually want multiple KVM modules. I'll plan on going that route. The very worst case scenario is that it just punts this conversation down to a possibile future. Dropping this patch and the previous prep patch won't meaningful affect the goals of this series, as only kvm_get_kvm_safe() and kvm_get_kvm() would need to be exposed outside of #ifdef __KVM__. Then we can figure out what to do with them if/when the whole multi-KVM thing comes along. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F20AFC4167B for ; Sat, 2 Dec 2023 00:53:01 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=P+OGwpec; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Shry372lWz3dBr for ; Sat, 2 Dec 2023 11:52:59 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=P+OGwpec; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--seanjc.bounces.google.com (client-ip=2607:f8b0:4864:20::44a; helo=mail-pf1-x44a.google.com; envelope-from=3rx9qzqykdig4qmzvos00sxq.o0yxuz6911o-pq7xu454.0bxmn4.03s@flex--seanjc.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Shrx33rqzz3c9y for ; Sat, 2 Dec 2023 11:52:05 +1100 (AEDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-6cddec65393so3790950b3a.1 for ; Fri, 01 Dec 2023 16:52:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701478317; x=1702083117; darn=lists.ozlabs.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=P+OGwpectnM214EeO19/RurBhNiaAuZ2PGnmPM3JLTNwLWczeM1p1atQ4nDPWTcBpp cxIsN8BIUNgWZ7qE78rh1rEm/RYN2n2EMUIfoiXrHVuRo+5dKLzaqNtXkfqLWF+amFR+ MAQOQoyh3OoypHzfRo8ozZZFqPMd8RCQOsXUCtgjfx//rpgSQGtkyqHFHc/NGmLX2iI8 p8MFsZA1mswthTqlhkNXWbwRZtA+LXwHp8SZHmfjvqZeTufpoNEJoZ5khVsBh8NGo3fa I+5iU/zdaTfjvwxez8mcak0g3ILr4czfggeavlZ23QNoRmyCPFoE0hJKsioounnC8FyR 6QXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701478317; x=1702083117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=WWUwqqCM3tWwpAhC+/PmO7m+qtwq7TyRGNTT0JIw3KZz57l8D1hKGcGx4KSDjsLgXp eeD97n9WUPSHdmeUDwdCn6c5qL65flkrd5kL8HML5e9GpZ5xG+pk8aD6ZOut+7u5Nl/d gX9hgLpVSGQmVy0Jda+7gABgdkhLEhmTTqbsFqD6tAjpukBhv3AACSQu+rJ1I/w0z3OP ua5cOax0HcrNm/yTUWjoxpcHYnsgCcxtZlT3yx8PFr4mVV8l6cYonoi0oZdfm+HSu+fe RDKJSmHeQaOdd887XimjrhHS14Nx9YLDxm7Kb1BmMVMszdufCg44i5R/d56z01Agt332 LbMQ== X-Gm-Message-State: AOJu0YyzqDcEERmBwcn4JD1Lgthik/aweL4HnBMZG2/NKQJKU7Rxy2Ws 5L4w9OMt/363NK2POKaC4yR3A/EHAHM= X-Google-Smtp-Source: AGHT+IEloABSTDcKm+41L5WpzazJRfhcCVpuae9X+15kKFthyi+8YQcZOmegDR+VoF+o36yu4wG4kUpnsoo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:5848:0:b0:5b8:fe99:152d with SMTP id i8-20020a635848000000b005b8fe99152dmr4028549pgm.7.1701478317322; Fri, 01 Dec 2023 16:51:57 -0800 (PST) Date: Fri, 1 Dec 2023 16:51:55 -0800 In-Reply-To: <20230918160258.GL13795@ziepe.ca> Mime-Version: 1.0 References: <20230916003118.2540661-1-seanjc@google.com> <20230916003118.2540661-6-seanjc@google.com> <20230918152110.GI13795@ziepe.ca> <20230918160258.GL13795@ziepe.ca> Message-ID: Subject: Re: [PATCH 05/26] vfio: KVM: Pass get/put helpers from KVM to VFIO, don't do circular lookup From: Sean Christopherson To: Jason Gunthorpe Content-Type: text/plain; charset="us-ascii" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: x86@kernel.org, kvm@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Dave Hansen , linux-kernel@vger.kernel.org, Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Janosch Frank , Harald Freudenberger , Marc Zyngier , Huacai Chen , Halil Pasic , Andrew Thornton , Ingo Molnar , Christian Borntraeger , Jason Herne , Albert Ou , Vasily Gorbik , Venkatesh Srinivas , Heiko Carstens , Arnaldo Carvalho de Melo , Alex Williamson , Borislav Petkov , And y Lutomirski , Paul Walmsley , kvmarm@lists.linux.dev, Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Tony Krowiak , Anish Ghulati , linux-mips@vger.kernel.org, Oliver Upton , linux-perf-users@vger.kernel.org, Palmer Dabbelt , kvm-riscv@lists.infradead.org, Anup Patel , Paolo Bonzini , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > On Mon, Sep 18, 2023 at 08:49:57AM -0700, Sean Christopherson wrote: > > On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > > > On Fri, Sep 15, 2023 at 05:30:57PM -0700, Sean Christopherson wrote: > > > > Explicitly pass KVM's get/put helpers to VFIO when attaching a VM to > > > > VFIO instead of having VFIO do a symbol lookup back into KVM. Having both > > > > KVM and VFIO do symbol lookups increases the overall complexity and places > > > > an unnecessary dependency on KVM (from VFIO) without adding any value. > > > > > > > > Signed-off-by: Sean Christopherson > > > > --- > > > > drivers/vfio/vfio.h | 2 ++ > > > > drivers/vfio/vfio_main.c | 74 +++++++++++++++++++--------------------- > > > > include/linux/vfio.h | 4 ++- > > > > virt/kvm/vfio.c | 9 +++-- > > > > 4 files changed, 47 insertions(+), 42 deletions(-) > > > > > > I don't mind this, but Christoph had disliked my prior attempt to do > > > this with function pointers.. > > > > > > The get can be inlined, IIRC, what about putting a pointer to the put > > > inside the kvm struct? > > > > That wouldn't allow us to achieve our goal, which is to hide the details of > > "struct kvm" from VFIO (and the rest of the kernel). > > > What's the objection to handing VFIO a function pointer? > > Hmm, looks like it was this thread: > > https://lore.kernel.org/r/0-v1-33906a626da1+16b0-vfio_kvm_no_group_jgg@nvidia.com > > Your rational looks a little better to me. > > > > The the normal kvm get/put don't have to exported symbols at all? > > > > The export of kvm_get_kvm_safe() can go away (I forgot to do that in this series), > > but kvm_get_kvm() will hang around as it's needed by KVM sub-modules (PPC and x86), > > KVMGT (x86), and drivers/s390/crypto/vfio_ap_ops.c (no idea what to call that beast). > > My thought would be to keep it as an inline, there should be some way > to do that without breaking your desire to hide the bulk of the kvm > struct content. Like put the refcount as the first element in the > struct and just don't ifdef it away?. That doesn't work because of the need to invoke kvm_destroy_vm() when the last reference is put, i.e. all of kvm_destroy_vm() would need to be inlined (LOL) or VFIO would need to do a symbol lookup on kvm_destroy_vm(), which puts back us at square one. There's one more wrinkle: this patch is buggy in that it doesn't ensure the liveliness of KVM-the-module, i.e. nothing prevents userspace from unloading kvm.ko while VFIO still holds a reference to a kvm structure, and so invoking ->put_kvm() could jump into freed code. To fix that, KVM would also need to pass along a module pointer :-( One thought would be to have vac.ko (tentative name), which is the "base" module that will hold the KVM/virtualization bits that need to be singletons, i.e. can't be per-KVM, provide the symbols needed for VFIO to manage references. But that just ends up moving the module reference trickiness into VAC+KVM, e.g. vac.ko would still need to be handed a function pointer in order to call back into the correct kvm.ko code. Hrm, but I suspect the vac.ko <=> kvm.ko interactions will need to deal with module shenanigans anyways, and the shenanigans would only affect crazy people like us that actually want multiple KVM modules. I'll plan on going that route. The very worst case scenario is that it just punts this conversation down to a possibile future. Dropping this patch and the previous prep patch won't meaningful affect the goals of this series, as only kvm_get_kvm_safe() and kvm_get_kvm() would need to be exposed outside of #ifdef __KVM__. Then we can figure out what to do with them if/when the whole multi-KVM thing comes along. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D633C4167B for ; Sat, 2 Dec 2023 00:52:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8+v0C6nSyjPgbeBcrHo9LP4Z4Bhn+hPcWuBZn+hfmUg=; b=c/taZm/5WPMkyKNys1jQbSo8JQ dj0vjNbBcddKdqVGPWT9W5DTmeFj1YbgKqJ84VriARKAOHJQ36dKU3uLWEawkJWg9hm5HZQ5nMgSE pcG7acqMR6CMZ2xWDzVEq9KbWLf410H+G13jkLrXONCmElznApzwqx7EHUnhJzI+MQCik5mIw2yJY UuYfrPZ2zwxl+zvQfMpUYeWPgMGacvIu8bb+iU8a/kJiJ2zU9OIREew560Mfr/lsqFv1RXIWRBFXH +Ut3Vab4KdKWDKJIdANWy3q9ON86wsr6T22jk9v1vOJaTAooX6X4rnyQ+Fwm11zSWN1y8Kn77Ej6N +zGY0xkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r9EEp-00EwdW-1x; Sat, 02 Dec 2023 00:52:07 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r9EEm-00EwcI-2y for linux-arm-kernel@lists.infradead.org; Sat, 02 Dec 2023 00:52:06 +0000 Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-6cddec65393so3790945b3a.1 for ; Fri, 01 Dec 2023 16:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701478317; x=1702083117; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=ssmpraemfge8cj9yx0bQe0olitdcprZSjL66pJDSdmW0aPZIfyKM0ZJp9V1tq2m8jj J1Ua1wVh0BVpFthTM9gr9XJBNyjlSD9Ay0NcihLc7oazbRDj84VHrGecjs/ZuYvbyAnM Irwag3DSPVoTeWXxyK+mpZdjdQUXh6FtVOCjwaRgdE8+X6ITCVI3DOrACYY4mY9DvgHd YzTzmsNR8OIBrxP2rimNl3luKd5BOiGlJCgH+urgFV4uBABJzgaRJHiXbCpaiuG6xbQ7 BywurV+G1BbrK6d3YrA1uUATEdddTUsB4QDF7q6w8TBFaPUbMiOh9cC0cliWt+7ajyYC kUQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701478317; x=1702083117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Qf5LyHZHuBq8ET8yZ518WK6d/Ohu8OSBvuXvyN934wU=; b=lfbR+BTcH9lflZIKZM1OeYSxQpeTDNXDxLt2zTqoPEtIKfNJJ9yb8GWU8QVsMbHSre NbzLaZ8dh4T/A2iy8R4dj3dVRxOv0JdBePL9qMGxLPl7Oh01kbz5UqidrjFFqYitb+IY mckQ/xGBS34F2lJ326jT7vdfCisJmtVNanPdkYuCwoiSWq3FFS4ZAtDTjP6ws3NQiUPw yYYnJ2wOlhWxMSb4eNC0Xq+UeHB5ZwRdApFfIZYuohFgHIhApW4dJpB8YAVWONznGUFB osTrdToUtDOQfr9aCwldBaKA3NYh5O5uolBOSzhuCQzxI5fXpmtajP4QcOOFOhqduUGC 2Kgg== X-Gm-Message-State: AOJu0YxUpAW0PgW4rQdhM0V+m+C5qRlffcZM8nUgZWOzbRnhZbUIRzlQ ritf7/HQQXp3RBbNEsMyFkh4ub3IL44= X-Google-Smtp-Source: AGHT+IEloABSTDcKm+41L5WpzazJRfhcCVpuae9X+15kKFthyi+8YQcZOmegDR+VoF+o36yu4wG4kUpnsoo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:5848:0:b0:5b8:fe99:152d with SMTP id i8-20020a635848000000b005b8fe99152dmr4028549pgm.7.1701478317322; Fri, 01 Dec 2023 16:51:57 -0800 (PST) Date: Fri, 1 Dec 2023 16:51:55 -0800 In-Reply-To: <20230918160258.GL13795@ziepe.ca> Mime-Version: 1.0 References: <20230916003118.2540661-1-seanjc@google.com> <20230916003118.2540661-6-seanjc@google.com> <20230918152110.GI13795@ziepe.ca> <20230918160258.GL13795@ziepe.ca> Message-ID: Subject: Re: [PATCH 05/26] vfio: KVM: Pass get/put helpers from KVM to VFIO, don't do circular lookup From: Sean Christopherson To: Jason Gunthorpe Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Peter Zijlstra , Arnaldo Carvalho de Melo , Paolo Bonzini , Tony Krowiak , Halil Pasic , Jason Herne , Harald Freudenberger , Alex Williamson , Andy Lutomirski , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Anish Ghulati , Venkatesh Srinivas , Andrew Thornton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231201_165204_984695_E575AAE6 X-CRM114-Status: GOOD ( 36.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > On Mon, Sep 18, 2023 at 08:49:57AM -0700, Sean Christopherson wrote: > > On Mon, Sep 18, 2023, Jason Gunthorpe wrote: > > > On Fri, Sep 15, 2023 at 05:30:57PM -0700, Sean Christopherson wrote: > > > > Explicitly pass KVM's get/put helpers to VFIO when attaching a VM to > > > > VFIO instead of having VFIO do a symbol lookup back into KVM. Having both > > > > KVM and VFIO do symbol lookups increases the overall complexity and places > > > > an unnecessary dependency on KVM (from VFIO) without adding any value. > > > > > > > > Signed-off-by: Sean Christopherson > > > > --- > > > > drivers/vfio/vfio.h | 2 ++ > > > > drivers/vfio/vfio_main.c | 74 +++++++++++++++++++--------------------- > > > > include/linux/vfio.h | 4 ++- > > > > virt/kvm/vfio.c | 9 +++-- > > > > 4 files changed, 47 insertions(+), 42 deletions(-) > > > > > > I don't mind this, but Christoph had disliked my prior attempt to do > > > this with function pointers.. > > > > > > The get can be inlined, IIRC, what about putting a pointer to the put > > > inside the kvm struct? > > > > That wouldn't allow us to achieve our goal, which is to hide the details of > > "struct kvm" from VFIO (and the rest of the kernel). > > > What's the objection to handing VFIO a function pointer? > > Hmm, looks like it was this thread: > > https://lore.kernel.org/r/0-v1-33906a626da1+16b0-vfio_kvm_no_group_jgg@nvidia.com > > Your rational looks a little better to me. > > > > The the normal kvm get/put don't have to exported symbols at all? > > > > The export of kvm_get_kvm_safe() can go away (I forgot to do that in this series), > > but kvm_get_kvm() will hang around as it's needed by KVM sub-modules (PPC and x86), > > KVMGT (x86), and drivers/s390/crypto/vfio_ap_ops.c (no idea what to call that beast). > > My thought would be to keep it as an inline, there should be some way > to do that without breaking your desire to hide the bulk of the kvm > struct content. Like put the refcount as the first element in the > struct and just don't ifdef it away?. That doesn't work because of the need to invoke kvm_destroy_vm() when the last reference is put, i.e. all of kvm_destroy_vm() would need to be inlined (LOL) or VFIO would need to do a symbol lookup on kvm_destroy_vm(), which puts back us at square one. There's one more wrinkle: this patch is buggy in that it doesn't ensure the liveliness of KVM-the-module, i.e. nothing prevents userspace from unloading kvm.ko while VFIO still holds a reference to a kvm structure, and so invoking ->put_kvm() could jump into freed code. To fix that, KVM would also need to pass along a module pointer :-( One thought would be to have vac.ko (tentative name), which is the "base" module that will hold the KVM/virtualization bits that need to be singletons, i.e. can't be per-KVM, provide the symbols needed for VFIO to manage references. But that just ends up moving the module reference trickiness into VAC+KVM, e.g. vac.ko would still need to be handed a function pointer in order to call back into the correct kvm.ko code. Hrm, but I suspect the vac.ko <=> kvm.ko interactions will need to deal with module shenanigans anyways, and the shenanigans would only affect crazy people like us that actually want multiple KVM modules. I'll plan on going that route. The very worst case scenario is that it just punts this conversation down to a possibile future. Dropping this patch and the previous prep patch won't meaningful affect the goals of this series, as only kvm_get_kvm_safe() and kvm_get_kvm() would need to be exposed outside of #ifdef __KVM__. Then we can figure out what to do with them if/when the whole multi-KVM thing comes along. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel