From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 88AA0C433F5 for ; Thu, 5 May 2022 14:53:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+MQHkvKEx8Qa3RW/1F2gyQ4QuzxM/0tbfI6pbLlmmQE=; b=tBvM9s3pbirWAb +yR5zqvYh7kxv5hBVWj8BOaraRmvyjKHAmKvYK0jeYnIsPA/5kINlT1JS1JtQVc46xYwYAs+TUE6d pMYzcRv4N2g3vjqJM6LptWjNPjGu2tk+Q0tqY/U7825MMgyKkDqIwD9mrVOoupnLrj+ekkBN5F9rb Ay9jJTUCInreztF+finQkj9m86LMiDpugKtx2nzQ7pQDi75bX+AHbei6iISfs+57FeYJwOMNuMf1P 5JNcsQ83lajufhD8B/2HiKluO4+24fMMhHnBDoKPiNYxYodXpnAX6HpcBdpD7kvso5HC+YpZLypop 38kuVJXlafczOXlHF2OA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmcpq-00GTDm-RY; Thu, 05 May 2022 14:52:07 +0000 Received: from us-smtp-delivery-74.mimecast.com ([170.10.129.74]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmcpn-00GTBw-Rc for linux-arm-kernel@lists.infradead.org; Thu, 05 May 2022 14:52:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651762320; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=uM+ofZNcfgbacCYmlCH29DRYVtctL+nEHY0EJrvY55Y=; b=Y5X4807VaLHqOiVVyOtsetiWL0P3nyBwMeqmnR3BNab6Egr6XOTncW/HscaOrbbUxvDAtW cuPxvlArbiAeE5/Ig+1oYVunHPBUgap0CdFy+VVkjcykmF01aEi8jk9SCj8JQAwTKIkK5P VXmKzTIg6vnma3aevyBnyv9dgzsrgP8= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-649-1F5KfVa3NYq9gyOvOCmFNw-1; Thu, 05 May 2022 10:51:57 -0400 X-MC-Unique: 1F5KfVa3NYq9gyOvOCmFNw-1 Received: by mail-wr1-f72.google.com with SMTP id v29-20020adfa1dd000000b0020ad932b7c0so1552283wrv.0 for ; Thu, 05 May 2022 07:51:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=uM+ofZNcfgbacCYmlCH29DRYVtctL+nEHY0EJrvY55Y=; b=t8ZyKuQDDYwaNb7VgqQo0+JkKcabLxaojteQA2tZPvP2uyz2WUCEI6ICgvnOL6XiJV dDvRHjbER9Nbtq8MsXsKkr7c7Lc9lY7QCOnJrwDFxwoGxB8RBsbn/+RJZjxQfdK5gVqc ehZxqjwBPBr8PjrHfu+a0lpfF6ZvLImXp5W4bSBOtiHY+nQ23JCCgkM+ppFsqJ0oH+r8 26JwBoCWxIuRQbNXxSA1FVWalUj4qlX0Nf/0iMbnGZze3SvFLkgVO7sbMbTJBx9rHA1i ApEL32REnNT+nFN+KUa013yxGTCSjWCvtJbmTMEELVkhPjOvrHyIUgkmHCI/q0V8PyUH T30Q== X-Gm-Message-State: AOAM532/IbTL7RlKu18XTtyfI33hos3lyu8lLrydjn4ffA5oFCwb1Tu3 UJaONt/dzj8efQyXUKR/R1GY0WtQm/mb5JDkyKI1nD9vO++srNgrDeEEEYUPpzoJ0r4k7TjLa/B xrgo0+1AgaBREbdeBgtZZg8vblXhjG3mLs4M= X-Received: by 2002:a1c:6a1a:0:b0:394:272e:5bdf with SMTP id f26-20020a1c6a1a000000b00394272e5bdfmr5347723wmc.55.1651762316171; Thu, 05 May 2022 07:51:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwzZ199d4B0hb5YxHFT/G5vw6e8Q1vc2C0FJNM5txW7lPwaFe4IrmxdRb6YfGAM2P9o3lQ5lw== X-Received: by 2002:a1c:6a1a:0:b0:394:272e:5bdf with SMTP id f26-20020a1c6a1a000000b00394272e5bdfmr5347690wmc.55.1651762315893; Thu, 05 May 2022 07:51:55 -0700 (PDT) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id 25-20020a05600c029900b003942a244ed1sm1524096wmk.22.2022.05.05.07.51.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 May 2022 07:51:55 -0700 (PDT) From: Vitaly Kuznetsov To: Mark Rutland Cc: "Guilherme G. Piccoli" , "Michael Kelley (LINUX)" , Marc Zyngier , Catalin Marinas , will Deacon , Russell King , Ard Biesheuvel , broonie@kernel.org, "linux-arm-kernel@lists.infradead.org" , linux-kernel , "linux-hyperv@vger.kernel.org" Subject: Re: Should arm64 have a custom crash shutdown handler? In-Reply-To: References: <427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com> <874k24igjf.wl-maz@kernel.org> <92645c41-96fd-2755-552f-133675721a24@igalia.com> <3bee47db-f771-b502-82a3-d6fac388aa89@igalia.com> <878rrg13zb.fsf@redhat.com> Date: Thu, 05 May 2022 16:51:54 +0200 Message-ID: <87y1zgyqut.fsf@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vkuznets@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220505_075204_025292_C94BC2F1 X-CRM114-Status: GOOD ( 41.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark Rutland writes: > On Thu, May 05, 2022 at 03:52:24PM +0200, Vitaly Kuznetsov wrote: >> "Guilherme G. Piccoli" writes: >> >> > On 05/05/2022 09:53, Mark Rutland wrote: >> >> [...] >> >> Looking at those, the cleanup work is all arch-specific. What exactly would we >> >> need to do on arm64, and why does it need to happen at that point specifically? >> >> On arm64 we don't expect as much paravirtualization as on x86, so it's not >> >> clear to me whether we need anything at all. >> >> >> >>> Anyway, the idea here was to gather a feedback on how "receptive" arm64 >> >>> community would be to allow such customization, appreciated your feedback =) >> >> >> >> ... and are you trying to do this for Hyper-V or just using that as an example? >> >> >> >> I think we're not going to be very receptive without a more concrete example of >> >> what you want. >> >> >> >> What exactly do *you* need, and *why*? Is that for Hyper-V or another hypervisor? >> >> >> >> Thanks >> >> Mark. >> > >> > Hi Mark, my plan would be doing that for Hyper-V - kind of the same >> > code, almost. For example, in hv_crash_handler() there is a stimer >> > clean-up and the vmbus unload - my understanding is that this same code >> > would need to run in arm64. Michael Kelley is CCed, he was discussing >> > with me in the panic notifiers thread and may elaborate more on the needs. >> > >> > But also (not related with my specific plan), I've seen KVM quiesce code >> > on x86 as well [see kvm_crash_shutdown() on arch/x86] , I'm not sure if >> > this is necessary for arm64 or if this already executing in some >> > abstracted form, I didn't dig deep - probably Vitaly is aware of that, >> > hence I've CCed him here. >> >> Speaking about the difference between reboot notifiers call chain and >> machine_ops.crash_shutdown for KVM/x86, the main difference is that >> reboot notifier is called on some CPU while the VM is fully functional, >> this way we may e.g. still use IPIs (see kvm_pv_reboot_notify() doing >> on_each_cpu()). When we're in a crash situation, >> machine_ops.crash_shutdown is called on the CPU which crashed. We can't >> count on IPIs still being functional so we do the very basic minimum so >> *this* CPU can boot kdump kernel. There's no guarantee other CPUs can >> still boot but normally we do kdump with 'nprocs=1'. > > Sure; IIUC the IPI problem doesn't apply to arm64, though, since that doesn't > use a PV mechanism (and practically speaking will either be GICv2 or GICv3). > This isn't really about PV: when the kernel is crashing, you have no idea what's going on on other CPUs, they may be crashing too, locked in a tight loop, ... so sending an IPI there to do some work and expecting it to report back is dangerous. >> For Hyper-V, the situation is similar: hv_crash_handler() intitiates >> VMbus unload on the crashing CPU only, there's no mechanism to do >> 'global' unload so other CPUs will likely not be able to connect Vmbus >> devices in kdump kernel but this should not be necessary. > > Given kdump is best-effort (and we can't rely on secondary CPUs even making it > into the kdump kernel), I also don't think that should be necessary. Yes, exactly. > >> There's a crash_kexec_post_notifiers mechanism which can be used instead >> but it's disabled by default so using machine_ops.crash_shutdown is >> better. > > Another option is to defer this to the kdump kernel. On arm64 at least, we know > if we're in a kdump kernel early on, and can reset some state based upon that. > > Looking at x86's hyperv_cleanup(), everything relevant to arm64 can be deferred > to just before the kdump kernel detects and initializes anything relating to > hyperv. So AFAICT we could have hyperv_init() check is_kdump_kernel() prior to > the first hypercall, and do the cleanup/reset there. In theory yes, it is possible to try sending CHANNELMSG_UNLOAD on kdump kernel boot and not upon crash, I don't remember if this approach was tried in the past. > > Maybe we need more data for the vmbus bits? ... if so it seems that could blow > up anyway when the first kernel was tearing down. Not sure I understood what you mean... From what I remember, there were issues with CHANNELMSG_UNLOAD handling on the Hyper-V host side in the past (it was taking *minutes* for the host to reply) but this is orthogonal to the fact that we need to do this cleanup so kdump kernel is able to connect to Vmbus devices again. -- Vitaly _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel