From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26CD7C432BE for ; Thu, 26 Aug 2021 17:50:38 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 887DF60EFE for ; Thu, 26 Aug 2021 17:50:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 887DF60EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0EBBC4B0BF; Thu, 26 Aug 2021 13:50:37 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EWND6bxY8Q6P; Thu, 26 Aug 2021 13:50:31 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id C9CBB4B0E5; Thu, 26 Aug 2021 13:50:31 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 114CB4B0BF for ; Thu, 26 Aug 2021 13:50:30 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 36eaLKpPZFNm for ; Thu, 26 Aug 2021 13:50:26 -0400 (EDT) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id E95DF4A1B0 for ; Thu, 26 Aug 2021 13:50:25 -0400 (EDT) Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E81256056B; Thu, 26 Aug 2021 17:50:24 +0000 (UTC) Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mJJWA-007P1z-SV; Thu, 26 Aug 2021 18:50:22 +0100 Date: Thu, 26 Aug 2021 18:50:22 +0100 Message-ID: <87zgt4nli9.wl-maz@kernel.org> From: Marc Zyngier To: Andre Przywara Subject: Re: [PATCH][kvmtool] virtio/pci: Correctly handle MSI-X masking while MSI-X is disabled In-Reply-To: <7c2abc6a-164a-b07e-a39e-756b83a98f0c@arm.com> References: <20210821120742.855712-1-maz@kernel.org> <20210823174833.05adee5d@slackpad.fritz.box> <87tujeq5ey.wl-maz@kernel.org> <87a6l5pmim.wl-maz@kernel.org> <878s0ppgff.wl-maz@kernel.org> <7c2abc6a-164a-b07e-a39e-756b83a98f0c@arm.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: andre.przywara@arm.com, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, alexandru.elisei@arm.com, tglx@linutronix.de, will@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: kernel-team@android.com, kvm@vger.kernel.org, Thomas Gleixner , Will Deacon , kvmarm@lists.cs.columbia.edu X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Thu, 26 Aug 2021 18:07:55 +0100, Andre Przywara wrote: > > On 8/25/21 6:44 PM, Marc Zyngier wrote: > > >> Actually, let me correct myself. I tested the wrong configuration (why > >> isn't --force-pci the bloody default in kvmtool?). > > I guess because there is no --force-mmio, and PCI on ARM was kind of > daunting back then ;-) I remember that. But maybe we should change that default... > >> Having dug further, it isn't the config space that causes problems, > >> but the programming of the MSI-X vectors. I'm starting to suspect the > >> layout of the MSI-X bar in kvmtool. > > > > OK, this is hilarious. Sort of. The MSI-X bar sizing is bonkers: you > > can't fit 33 MSIs there (33 being the number of MSI-X that kvmtool > > advertises), and you will have notionally overwritten the PBA as > > well. Amusingly, the last write ends up being misdecoded as a config > > space access... > > Ah, very good find indeed, many thanks for the debugging! > > I am only halfway through the code by now, but wouldn't you need to > adjust the PBA offset in the MSIX capability as well? This is still > stuck at that (misnamed) PCI_IO_SIZE, in > virtio/pci.c:virtio_pci__init(): vpci->pci_hdr.msix.pba_offset = > And IIUC this has to match the decoding in virtio_pci__msix_mmio_callback(). Indeed. I had this: Region 2: Memory at 50000800 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] MSI-X: Enable+ Count=33 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00000100 and with the PBA fix, I get: Region 2: Memory at 50000800 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] MSI-X: Enable+ Count=33 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00000210 which looks more sane. But man, this code is crufty as hell. I'll post an updated patch later, once I've cleaned up a couple more things. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm