From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E61ABC6FD1F for ; Tue, 19 Mar 2024 17:54:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XckRnW1YKyNjJ6Y2iWb9xqGIBtRnmIol7uOhz/ME47I=; b=V6O5uuqbzgJIR1 2TdMBDtAPMi8ZHH5RJZPK1wTxfTIR1sdHQGqZcpeU4OAnJicJN3xt2YmM9MpUu7y4qjFnQAUDMbn1 0vCNqK+nU/yNzd+IK888lpGFinW1QZSsPYawHnpGuLnLRv0ykkaD+9DfkYMkRSH44xNq4pK9asAwe 6visgFUJljEfFU2UaIHw/JhAKrHyZCnuf2jhn6oafFcnRFVJ53gDeP4s2LQT13b+Vowz9GKtirohp Wda2dc0e0OB3D+g2BwnZ1QMi8c7u+QL1iiukKsbdYxDMvurWHrqoo7YPIyaercoLpcQ3rq7ltTx9V xAx5YmYKnlfDhr2h63TA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rmdfU-0000000Dgxr-3wjg; Tue, 19 Mar 2024 17:54:32 +0000 Received: from mail-oa1-x29.google.com ([2001:4860:4864:20::29]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rmdfQ-0000000Dgvt-4A3z for linux-arm-kernel@lists.infradead.org; Tue, 19 Mar 2024 17:54:31 +0000 Received: by mail-oa1-x29.google.com with SMTP id 586e51a60fabf-222b6a05bb1so2232116fac.3 for ; Tue, 19 Mar 2024 10:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1710870866; x=1711475666; darn=lists.infradead.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=B9HuiwQg7X+0lbII8qyF6ES+SfD7F1pHs2MmLusxi84=; b=A++yuNxKhxFkd8XKHVyysXt7hGIqEorCRRBKjScYLevEMrIZtHViHkauWYxOTSn7XK jSJ/mGrD+GW4ngr06eUi+9V7bmaU1EN+U2qzBgtxKQkim9fNKb8r4LA6DzGNggClfSKq ybDlFv5QkNQDFwKjPogjIutRa+8gDttYyU+eWNSlX5MF7G4BaAPJ/FowZq1McdMgsCkd UXSMtCkuwB6AKfxeF4W7Q2tfkc+FmCwHRytE/B47rlTC4Ukh8FlYHZsYmcmS+lLUJISP kbA+zcfAiYvgl3Cnhp1snMT80uDKuSzreMq+1WV++udcDCXnp9XNsw1zz3eaV4jbaN+n PKiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710870866; x=1711475666; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=B9HuiwQg7X+0lbII8qyF6ES+SfD7F1pHs2MmLusxi84=; b=wGvEkYz5WewBeLXSMXqDxPFvOS0xLjGHuvTnxBHrQwmqjMPcEpUnHlrg0gdgAYF6tt PFiao7GAhlg3zLNaSPfb5UCS8pL15g83A5pUIL/AisKfGMlhJ50f+2+1f4lz+C1vcnbj Asev+0A8QKplT6Cmbdaxu6gcnAIaYbZHRAXlV1W8N7h9Xh8we8+qaKSl9ie32fsTb0ss TS49v/AXYUJQHaNanWPZAWlQQ+igbktS3+rCMn1k4zrx/em2hDx1w08HSWO1gl9mZyMo Q6YIscgCdnrPFZycWQY4mf1gZdLMKwXDhBrkaQRpLRahNmEzKkpACPgUMEo9/q5o1O7E kEzQ== X-Forwarded-Encrypted: i=1; AJvYcCV6lhol3h9UE2BWE/FAyufJhiSbrUk8hmLkujzf99723v7tcJdC1h3+edWDMyDuMhtLes5kVL5cpzMnjVfXGd4Sv/VzLkdU8tDBsdl6lLmQ2/sdZUM= X-Gm-Message-State: AOJu0YzzI1ycYDq48KRnt3EDUWgAA6+gnjK54WMNnFKBnm2zOoK9Uok0 mJIy7nzPd0/gR/CO9XtL8EVMrEhQrM9pfEaw74nSjgN/BpVVZuDVwShR/nbRHDHgtbGgbUq93FC gj3U= X-Google-Smtp-Source: AGHT+IEd8KsCy6GqIl+1n0Sol2b9f0f/25EaZRJ80XDCZoyr9TJbEhweqbC03Izk0lN9FbayxSp4fw== X-Received: by 2002:a05:6870:524f:b0:221:bf42:cf76 with SMTP id o15-20020a056870524f00b00221bf42cf76mr16367906oai.10.1710870866557; Tue, 19 Mar 2024 10:54:26 -0700 (PDT) Received: from ziepe.ca ([12.97.180.36]) by smtp.gmail.com with ESMTPSA id gh11-20020a056638698b00b00477716fcbb8sm2429986jab.40.2024.03.19.10.54.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 10:54:25 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1rmdbD-001kiW-EG; Tue, 19 Mar 2024 14:50:07 -0300 Date: Tue, 19 Mar 2024 14:50:07 -0300 From: Jason Gunthorpe To: Will Deacon Cc: Robin Murphy , Tyler Hicks , Jerry Snitselaar , linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Dexuan Cui , Easwar Hariharan Subject: Re: Why is the ARM SMMU v1/v2 put into bypass mode on kexec? Message-ID: <20240319175007.GC66976@ziepe.ca> References: <120d0dec-450f-41f8-9e05-fd763e84f6dd@arm.com> <20240319154756.GB2901@willie-the-truck> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240319154756.GB2901@willie-the-truck> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240319_105429_193210_1C30F4F0 X-CRM114-Status: GOOD ( 25.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Mar 19, 2024 at 03:47:56PM +0000, Will Deacon wrote: > Right, it's hard to win if DMA-active devices weren't quiesced properly > by the outgoing kernel. Either the SMMU was left in abort (leading to the > problems you list above) or the SMMU is left in bypass (leading to possible > data corruption). Which is better? For whatever reason (and I really don't like this design) alot of work was done on x86 so that device continues to work as-was right up until the crash kernel does the first DMA operation. Including having the crash kernel non disruptively inherit and retain the IOMMU configuration. (eg see translation_pre_enabled() stuff in intel driver) I think the idea was that the crash kernel driver will recover control of the device prior to trying to do DMA. Devices without a driver or devices that are not operated by the crash kernel just keep going as they were. In general practice this is unworkable as some devices can't be recovered without doing DMA in the first place creating a catch-22. So now lots of devices use their shutdown handler to quiet the device before handing over to the crash kernel. I think this emerged as some 'small work' to try and make crash kernels functional at all. Implementing every shutdown handler would be pretty hard, but many (?) devices seem to work OK if the crash kernel drivers runs for a bit before destroying their DMA setup. We don't trigger weird platform crashes or anything due to failing DMA operations either. Now we have all kinds of infrastructure and deployed crash kernels that have this assumption baked in. :( It sure would be nice to not spread this full complexity to ARM. If the original kernel could signal to the crash kernel that specific devices are quieted and then the crash kernel could simply ignore unquieted devices and set the IOMMU to abort them and don't allow any crash drivers to attach. (or maybe FLR them?) If someone wants a device to be usuable in the crash kernel then the original kernel needs to implement the shutdown handler. Regardless, I think if your goal is to support crash kernels then you have to do at least a bit of the x86 'keep the iommu unchanged'. The iommu shutdown should do less like x86 does and the iommu startup should detect the special case and try to atomic switch to the new STE table that aborts unquieted devices. Booting a non-crash OS is a different matter and in that case you really want every bit of HW put back to a clean "just booted" state, and arguably it can't work unless the original kernel implements all the shutdown handlers... I don't know if x86 kexec actually support this, it looks like it only works on Linux OS and things like the Linux iommu driver have code to support the crash focused hand over even in non crash cases. Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel