From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A11EC432BE for ; Thu, 26 Aug 2021 17:50:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B4A260BD3 for ; Thu, 26 Aug 2021 17:50:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233530AbhHZRvM (ORCPT ); Thu, 26 Aug 2021 13:51:12 -0400 Received: from mail.kernel.org ([198.145.29.99]:40872 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230306AbhHZRvM (ORCPT ); Thu, 26 Aug 2021 13:51:12 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E81256056B; Thu, 26 Aug 2021 17:50:24 +0000 (UTC) Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mJJWA-007P1z-SV; Thu, 26 Aug 2021 18:50:22 +0100 Date: Thu, 26 Aug 2021 18:50:22 +0100 Message-ID: <87zgt4nli9.wl-maz@kernel.org> From: Marc Zyngier To: Andre Przywara Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, Alexandru Elisei , Thomas Gleixner , Will Deacon Subject: Re: [PATCH][kvmtool] virtio/pci: Correctly handle MSI-X masking while MSI-X is disabled In-Reply-To: <7c2abc6a-164a-b07e-a39e-756b83a98f0c@arm.com> References: <20210821120742.855712-1-maz@kernel.org> <20210823174833.05adee5d@slackpad.fritz.box> <87tujeq5ey.wl-maz@kernel.org> <87a6l5pmim.wl-maz@kernel.org> <878s0ppgff.wl-maz@kernel.org> <7c2abc6a-164a-b07e-a39e-756b83a98f0c@arm.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: andre.przywara@arm.com, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, alexandru.elisei@arm.com, tglx@linutronix.de, will@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Thu, 26 Aug 2021 18:07:55 +0100, Andre Przywara wrote: > > On 8/25/21 6:44 PM, Marc Zyngier wrote: > > >> Actually, let me correct myself. I tested the wrong configuration (why > >> isn't --force-pci the bloody default in kvmtool?). > > I guess because there is no --force-mmio, and PCI on ARM was kind of > daunting back then ;-) I remember that. But maybe we should change that default... > >> Having dug further, it isn't the config space that causes problems, > >> but the programming of the MSI-X vectors. I'm starting to suspect the > >> layout of the MSI-X bar in kvmtool. > > > > OK, this is hilarious. Sort of. The MSI-X bar sizing is bonkers: you > > can't fit 33 MSIs there (33 being the number of MSI-X that kvmtool > > advertises), and you will have notionally overwritten the PBA as > > well. Amusingly, the last write ends up being misdecoded as a config > > space access... > > Ah, very good find indeed, many thanks for the debugging! > > I am only halfway through the code by now, but wouldn't you need to > adjust the PBA offset in the MSIX capability as well? This is still > stuck at that (misnamed) PCI_IO_SIZE, in > virtio/pci.c:virtio_pci__init(): vpci->pci_hdr.msix.pba_offset = > And IIUC this has to match the decoding in virtio_pci__msix_mmio_callback(). Indeed. I had this: Region 2: Memory at 50000800 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] MSI-X: Enable+ Count=33 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00000100 and with the PBA fix, I get: Region 2: Memory at 50000800 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] MSI-X: Enable+ Count=33 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00000210 which looks more sane. But man, this code is crufty as hell. I'll post an updated patch later, once I've cleaned up a couple more things. Thanks, M. -- Without deviation from the norm, progress is not possible.