From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4817C3271E for ; Mon, 8 Jul 2024 15:40:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=ko4nEiDfhlR7fhraWGQD/gHhjHGleJUcF9JQDJhaqVg=; b=BXNsOKhXWqvpVcmjlsxnQduB2V Z3gn13mjYYqhrfq+5h/nQt/inNitFnWrtkG2YjmDVUL0kUO1m1OWAeVT4Yq3gH7u0th3Gk90Nv/CA DkHplCL1A5NDus/joe/bUccFzCWOLSlESvkzUGe7uq38eiUpaQf35oOs+NvTfIL0D1YKNoznmHAA3 AJyr7uVS42Q0yNCbJpM3Zew4YohyAKbMlEOUl++w9iH5CZCyASjaoCweA2z4CjRFYkgCZtum4z2DX JpuGgDyJcrB2m1+8PMX0upm3X3/wNWPIpgYFfhEHjQfrKwu0pDTRohMdY19yy7k4+Q1i8Jx3fbe44 BnHDS5Ag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sQqT4-00000004Ft9-3ltS; Mon, 08 Jul 2024 15:39:54 +0000 Received: from mail-oo1-xc32.google.com ([2607:f8b0:4864:20::c32]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sQqSp-00000004FpG-3Xgd for linux-arm-kernel@lists.infradead.org; Mon, 08 Jul 2024 15:39:41 +0000 Received: by mail-oo1-xc32.google.com with SMTP id 006d021491bc7-5b53bb4bebaso2364321eaf.0 for ; Mon, 08 Jul 2024 08:39:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1720453178; x=1721057978; darn=lists.infradead.org; h=content-transfer-encoding:content-disposition:mime-version :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=ko4nEiDfhlR7fhraWGQD/gHhjHGleJUcF9JQDJhaqVg=; b=Eil7LoGm+Ws0Mq0ZUXHIwthlTCQGRyDsqKR74N3kk6RrYYmBkzAzua/m38Ms8Z/oQ5 RZkj4u/6rl5AiaYJQ/ArKvVNdvDVtb9nMQmXBxkDMHAuV/iaAHOYxhMU/YzsRrWf170s R8xA+6w8c757G6wVqW2y9l4cvvymVbLBi49Qphk/Hpr79l7BKSkyBa2RLcu8geTEfdJf szPtzC8UP0ylmpe2GY1TbXqajJG1Ca3tXls+2/KaGAp8xelSawSS0MeUr6J35Xs2kTJR ys7bq7iUz3nYoAM+RI52XGPk63UrD3z0ma4cSYid74uQ81zIg4BkS4oPWgxn73ZQCjej o5CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720453178; x=1721057978; h=content-transfer-encoding:content-disposition:mime-version :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ko4nEiDfhlR7fhraWGQD/gHhjHGleJUcF9JQDJhaqVg=; b=jhscKxDgS0Ki3+2TZDIJNOEUNSryAgZdMWE8HlXk4FovPekhOcApmhuwHyr/xKydVB ObgBP7s9QRDKxapXwbXDw2pUoWZ6dTguWKENWtfd7lodDPK1XCvbbL5R5b0+CB+MeWdF R1u26x0S3EJyaeWSFt0ExWX1jbtfPMmN+2ZUoA9ubCEGDDbM07/zzR5y03hHFsb63wuq j0EEvpQvxOGl6BdT3gkxCAnSnOiOSzl4OtFeOHfCrah405SAhcMZGDL4pn6f9Rpiuu/N P5yR6gPvkdjj8JPL2mrNkr/d0d/VduNRzw1f4YuGvTxOcADeOlYvHGQ1cuRxgTcqj9MU maTQ== X-Gm-Message-State: AOJu0YzM0XbO1qPFe6UQ4MiBFFPuynb52kYATx0udPN3Bupx/yUPcvSb ajV/8ueU8ob2nb5yAAHThOkZt1xLF8bZBlFKNBUnaD7/kXYSuwIzjuBEF2a4iw== X-Google-Smtp-Source: AGHT+IHZ+X32NksTb54ym15qqLdWcT2/2e6/hNZhPy7Iz2Eghsby8ddGDc0YUlTOyQwI2Yx4TvgURA== X-Received: by 2002:a05:6870:9690:b0:24f:c31a:5c29 with SMTP id 586e51a60fabf-25e2bf1eebbmr10662019fac.43.1720453177758; Mon, 08 Jul 2024 08:39:37 -0700 (PDT) Received: from thinkpad ([120.56.193.130]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70b1d15b336sm4760838b3a.198.2024.07.08.08.39.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Jul 2024 08:39:37 -0700 (PDT) Date: Mon, 8 Jul 2024 21:09:33 +0530 From: Manivannan Sadhasivam To: maz@kernel.org, tglx@linutronix.de Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: MSIs not freed in GICv3 ITS driver Message-ID: <20240708153933.GC5745@thinkpad> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240708_083939_907349_7854121B X-CRM114-Status: GOOD ( 14.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Marc, Thomas, I'm seeing a weird behavior with GICv3 ITS driver while allocating MSIs from PCIe devices. When the PCIe driver (I'm using virtio_pci_common.c) tries to allocate non power of 2 MSIs (like 3), then the GICv3 MSI driver always rounds the MSI count to power of 2 to find the order. In this case, the order becomes 2 in its_alloc_device_irq(). So 4 entries are allocated by bitmap_find_free_region(). But since the PCIe driver has only requested 3 MSIs, its_irq_domain_alloc() will only allocate 3 MSIs, leaving one bitmap entry unused. And when the driver frees the MSIs using pci_free_irq_vectors(), only 3 allocated MSIs were freed and their bitmap entries were also released. But the entry for the additional bitmap was never released. Due to this, its_free_device() was also never called, resulting in the ITS device not getting freed. So when the PCIe driver tries to request the MSIs again (PCIe device being removed and inserted back), because the ITS device was not freed previously, MSIs were again requested for the same ITS device. And due to the stale bitmap entry, the ITS driver refuses to allocate 4 MSIs as only 3 bitmap entries were available. This forces the PCIe driver to reduce the MSI count, which is sub optimal. This behavior might be applicable to other irqchip drivers handling MSI as well. I want to know if this behavior is already known with MSI and irqchip drivers? For fixing this issue, the PCIe drivers could always request MSIs of power of 2, and use a dummy MSI handler for the extra number of MSIs allocated. This could also be done in the generic MSI driver itself to avoid changes in the PCIe drivers. But I wouldn't say it is the best possible fix. Is there any other way to address this issue? Or am I missing something completely? - Mani -- மணிவண்ணன் சதாசிவம்