From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4635C433ED for ; Fri, 7 May 2021 09:03:57 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 0968061457 for ; Fri, 7 May 2021 09:03:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0968061457 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6C1CE4B513; Fri, 7 May 2021 05:03:56 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ajw7FeWP6DuY; Fri, 7 May 2021 05:03:55 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 597AD4B544; Fri, 7 May 2021 05:03:55 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id AD0A24B542 for ; Fri, 7 May 2021 05:03:53 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fhaZIBcyzRQ8 for ; Fri, 7 May 2021 05:03:52 -0400 (EDT) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 977914B525 for ; Fri, 7 May 2021 05:03:52 -0400 (EDT) Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 757386144A; Fri, 7 May 2021 09:03:51 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1lewOj-00BQgL-9i; Fri, 07 May 2021 10:03:49 +0100 Date: Fri, 07 May 2021 10:03:48 +0100 Message-ID: <87k0oaq5jf.wl-maz@kernel.org> From: Marc Zyngier To: Shaokun Zhang Subject: Re: Question on guest enable msi fail when using GICv4/4.1 In-Reply-To: <3a2c66d6-6ca0-8478-d24b-61e8e3241b20@hisilicon.com> References: <3a2c66d6-6ca0-8478-d24b-61e8e3241b20@hisilicon.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: zhangshaokun@hisilicon.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-pci@vger.kernel.org, alex.williamson@redhat.com, cohuck@redhat.com, tangnianyao@huawei.com, bhelgaas@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: kvm@vger.kernel.org, linux-pci@vger.kernel.org, Cornelia Huck , Alex Williamson , Bjorn Helgaas , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Fri, 07 May 2021 06:57:04 +0100, Shaokun Zhang wrote: > > [This letter comes from Nianyao Tang] > > Hi, > > Using GICv4/4.1 and msi capability, guest vf driver requires 3 > vectors and enable msi, will lead to guest stuck. Stuck how? > Qemu gets number of interrupts from Multiple Message Capable field > set by guest. This field is aligned to a power of 2(if a function > requires 3 vectors, it initializes it to 2). So I guess this is a MultiMSI device with 4 vectors, right? > However, guest driver just sends 3 mapi-cmd to vits and 3 ite > entries is recorded in host. Vfio initializes msi interrupts using > the number of interrupts 4 provide by qemu. When it comes to the > 4th msi without ite in vits, in irq_bypass_register_producer, > producer and consumer will __connect fail, due to find_ite fail, and > do not resume guest. Let me rephrase this to check that I understand it: - The device has 4 vectors - The guest only create mappings for 3 of them - VFIO calls kvm_vgic_v4_set_forwarding() for each vector - KVM doesn't have a mapping for the 4th vector and returns an error - VFIO disable this 4th vector Is that correct? If yes, I don't understand why that impacts the guest at all. From what I can see, vfio_msi_set_vector_signal() just prints a message on the console and carries on. > Do we support this case, Guest function using msi interrupts number > not aligned to a power of 2? Or qemu should provide correct msi > interrupts number? QEMU cannot know how many vectors are in use, and the guest is free to issue mappings for the exact number of vectors it wants to service. Please describe what breaks the guest here. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EC0CC433B4 for ; Fri, 7 May 2021 09:03:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1021C6144F for ; Fri, 7 May 2021 09:03:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231745AbhEGJEv (ORCPT ); Fri, 7 May 2021 05:04:51 -0400 Received: from mail.kernel.org ([198.145.29.99]:60546 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230108AbhEGJEu (ORCPT ); Fri, 7 May 2021 05:04:50 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 757386144A; Fri, 7 May 2021 09:03:51 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1lewOj-00BQgL-9i; Fri, 07 May 2021 10:03:49 +0100 Date: Fri, 07 May 2021 10:03:48 +0100 Message-ID: <87k0oaq5jf.wl-maz@kernel.org> From: Marc Zyngier To: Shaokun Zhang Cc: , , , , Alex Williamson , Cornelia Huck , Nianyao Tang , Bjorn Helgaas Subject: Re: Question on guest enable msi fail when using GICv4/4.1 In-Reply-To: <3a2c66d6-6ca0-8478-d24b-61e8e3241b20@hisilicon.com> References: <3a2c66d6-6ca0-8478-d24b-61e8e3241b20@hisilicon.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: zhangshaokun@hisilicon.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-pci@vger.kernel.org, alex.williamson@redhat.com, cohuck@redhat.com, tangnianyao@huawei.com, bhelgaas@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Fri, 07 May 2021 06:57:04 +0100, Shaokun Zhang wrote: > > [This letter comes from Nianyao Tang] > > Hi, > > Using GICv4/4.1 and msi capability, guest vf driver requires 3 > vectors and enable msi, will lead to guest stuck. Stuck how? > Qemu gets number of interrupts from Multiple Message Capable field > set by guest. This field is aligned to a power of 2(if a function > requires 3 vectors, it initializes it to 2). So I guess this is a MultiMSI device with 4 vectors, right? > However, guest driver just sends 3 mapi-cmd to vits and 3 ite > entries is recorded in host. Vfio initializes msi interrupts using > the number of interrupts 4 provide by qemu. When it comes to the > 4th msi without ite in vits, in irq_bypass_register_producer, > producer and consumer will __connect fail, due to find_ite fail, and > do not resume guest. Let me rephrase this to check that I understand it: - The device has 4 vectors - The guest only create mappings for 3 of them - VFIO calls kvm_vgic_v4_set_forwarding() for each vector - KVM doesn't have a mapping for the 4th vector and returns an error - VFIO disable this 4th vector Is that correct? If yes, I don't understand why that impacts the guest at all. From what I can see, vfio_msi_set_vector_signal() just prints a message on the console and carries on. > Do we support this case, Guest function using msi interrupts number > not aligned to a power of 2? Or qemu should provide correct msi > interrupts number? QEMU cannot know how many vectors are in use, and the guest is free to issue mappings for the exact number of vectors it wants to service. Please describe what breaks the guest here. Thanks, M. -- Without deviation from the norm, progress is not possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E92FC433ED for ; Fri, 7 May 2021 09:06:40 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A959F60200 for ; Fri, 7 May 2021 09:06:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A959F60200 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=p5XIhMbLBtScAwbi1+RFWWJ6WqzgxTEd4TCLc05e2AU=; b=jLf8CLuEjhY/EO2kTAzO2ve6j KBOa+ohSIlWNp34rUI5ZopSrHXAnIIFdBmgvTnXZYywKkA2e5x8I7aqk+8f+rAZYfsSYaIUxK0L7a 38DUl5P8eE8HoiNm0M0+OrO85JtL4qJ9fkbfVgUR2obGtjwB4AMMRYE+ubhmPX4DmsgMnZLa7T5Ju u1QhBnbEik9yI96P/K9OskCCRjgusPHHrSuo/Ni/MkRxyC+ZqQbZnWIAEvh6wbLUkjxw+a/C6cjIW TS7HAoAdv7Z5m/OKmhdUXZyspE9S8QhIJ3W28GTlnbOy7qGDlLAZuQIrA53aWKB15Lb5jYpC0rdiF aacbtsVzg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lewPf-006ZKY-DT; Fri, 07 May 2021 09:04:47 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lewOo-006Yv3-Ca for linux-arm-kernel@desiato.infradead.org; Fri, 07 May 2021 09:03:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=7mXV7eH+nnZEgInzHrbUAsa64HdwS45Xq36tUE6n0k8=; b=cfYyBGrSG8bkdHD4oPxoplBfzS V8zUOPxbCgkgCLc6lXaAOao0SEHu6ibmHAdRNnwxFFC/vkIaE8MD5ILPtZLoeHHPbzG77Jv3zeVtG btIOZfUBFkhFaMRXYXaVhunMB6TDuSKTHow4y1pfbrKkEYUYYCBsUj6QCQfmnmoC2e4LPTF6/pRJG z5E/TgjEU2zi3odOMEsvWRT4F5Xv6KKZWdtg6EJLM8ufioxSeK7PITZfMbEYL/gA46TFg8e/eNBWg x5iF7TPN3NoK0HtE3avISBslufhOx/+a+IgM+n9usoysBwvmXkCylhe+lE9LPSRafZxFQMO1L8B/r u+TgsmDg==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lewOl-006ib0-TF for linux-arm-kernel@lists.infradead.org; Fri, 07 May 2021 09:03:53 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 757386144A; Fri, 7 May 2021 09:03:51 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1lewOj-00BQgL-9i; Fri, 07 May 2021 10:03:49 +0100 Date: Fri, 07 May 2021 10:03:48 +0100 Message-ID: <87k0oaq5jf.wl-maz@kernel.org> From: Marc Zyngier To: Shaokun Zhang Cc: , , , , Alex Williamson , Cornelia Huck , Nianyao Tang , Bjorn Helgaas Subject: Re: Question on guest enable msi fail when using GICv4/4.1 In-Reply-To: <3a2c66d6-6ca0-8478-d24b-61e8e3241b20@hisilicon.com> References: <3a2c66d6-6ca0-8478-d24b-61e8e3241b20@hisilicon.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: zhangshaokun@hisilicon.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-pci@vger.kernel.org, alex.williamson@redhat.com, cohuck@redhat.com, tangnianyao@huawei.com, bhelgaas@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210507_020352_002168_70447892 X-CRM114-Status: GOOD ( 24.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 07 May 2021 06:57:04 +0100, Shaokun Zhang wrote: > > [This letter comes from Nianyao Tang] > > Hi, > > Using GICv4/4.1 and msi capability, guest vf driver requires 3 > vectors and enable msi, will lead to guest stuck. Stuck how? > Qemu gets number of interrupts from Multiple Message Capable field > set by guest. This field is aligned to a power of 2(if a function > requires 3 vectors, it initializes it to 2). So I guess this is a MultiMSI device with 4 vectors, right? > However, guest driver just sends 3 mapi-cmd to vits and 3 ite > entries is recorded in host. Vfio initializes msi interrupts using > the number of interrupts 4 provide by qemu. When it comes to the > 4th msi without ite in vits, in irq_bypass_register_producer, > producer and consumer will __connect fail, due to find_ite fail, and > do not resume guest. Let me rephrase this to check that I understand it: - The device has 4 vectors - The guest only create mappings for 3 of them - VFIO calls kvm_vgic_v4_set_forwarding() for each vector - KVM doesn't have a mapping for the 4th vector and returns an error - VFIO disable this 4th vector Is that correct? If yes, I don't understand why that impacts the guest at all. From what I can see, vfio_msi_set_vector_signal() just prints a message on the console and carries on. > Do we support this case, Guest function using msi interrupts number > not aligned to a power of 2? Or qemu should provide correct msi > interrupts number? QEMU cannot know how many vectors are in use, and the guest is free to issue mappings for the exact number of vectors it wants to service. Please describe what breaks the guest here. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel