From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9352628DDA for ; Thu, 18 Jan 2024 14:22:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705587768; cv=none; b=BOnXYNxo3htmdtbE1axrtvFr30o7ZhYL9VylC3UBLNSxlLLOPiP8IFvkl/0PdNnW2u69GlqFThpgkNfP65eBODCY9WxPGkm1+vAQJn0yLTRmcVr1e6l2UU3h/8rlYbg0Zrb7wF9lo89t3SHQ2UID2xIJ2/mapOcZH16JrCNzX7w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705587768; c=relaxed/simple; bh=49cNhVsuBMgtJHArJcV1m7agcOAtGe87ch9+R4pb5zg=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received: Received:Date:From:To:Cc:Subject:Message-ID:References: MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To; b=K+zW8I6f9ivi7U6F5XTrV31DSefc9ewX+STDy4Qu373H+T+k4gFL3GSbEDDptHZpvkv+ETzGb2xFVN8EHfcW/oKcyr0IU9/BTAXe6N8THTt6DSNc5gVeD20nM0AFjkdyYKSNC2Iv/OnqA6vZkLZaQkZscurMj0O3AvBDPC7z7xs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=cJMwkt0X; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="cJMwkt0X" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3bbc649c275so5619067b6e.0 for ; Thu, 18 Jan 2024 06:22:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1705587765; x=1706192565; darn=lists.linux.dev; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=MQmQ57jhPNOOjCerNM6q5F9m10YHH8SVIWhIL9dMZTE=; b=cJMwkt0XMZtpRUS/H3QmmzgQcoKAUl3/w1KqlyLivkQRkM7pRbk20ae4gceUvv5+SW tY1UWv2cfhG54Q2hU7pNDsg7cjtByKDxASzRy7kpeAQFH5WUWp4yxi6JT1XU6CkwU5+s 32qrbYWEul8DdNjsqpUev2GPv1uaum/YURnIIB1pCdn2bdMSbcKUfFNcE7unUDTOO6qy nPUOXCEQP5l8U8vLAlvgOHHwUpYqr+j+Wdqy3ZhxZbYeHKXwKCJRKmLtx0JZPYse0GkW SPmT3cvouSOJYnKZQOBKV+71M0i8rUYe0Q1SFIATwZb/dVUYTzovO7Kj2TeX1XxQuWs5 yv6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705587765; x=1706192565; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MQmQ57jhPNOOjCerNM6q5F9m10YHH8SVIWhIL9dMZTE=; b=oZefSZkpJ5l1Kv4XxRxnAGE6BQ/GKkqAy5kC+Wk7UC4EylzDLVpDGeUNiOmxH4TB4e T7X9k3KOeb2ouJ/6d0vTM+EyYKEA6uF/l35c9mCnIo74KEBaKWpIUJtJjcpPshWV000d ueaw4ZP9OP20JaL6Uh2ox3ugsXAwY7lbdVZUOldl1wI2tiS2OWkBczNyudb8TldGEPI5 DjZoJcrI3AuGmfmUa50SUgDtJxFtDmEz/bXzcWrpaHKNzTAmfQmAMWRspaqCi1oMinbT u20+h7/1chDRmF/1LbxooM5+FQleJ8E6TZ6deC+ZBXEm8FEdjfCysgM0k65zKEAvkjvh LCCw== X-Gm-Message-State: AOJu0YyceSST/ZdIkAAaX9IlMicQVSoeIMPwmC0VcLinwFMbm5p6ovLb mGpsvIR+xcTPA2ojdAqfDzutDxK4QKCMUNEuDk6Yenc00CXKoOaluRXlZu385uQ= X-Google-Smtp-Source: AGHT+IEAZqGHugP6SSHGhmnB2UTe2rde66vWr8xnTZZd44KtjGiA2gbMs46TD4AdmZ56Y+ynbP+VDg== X-Received: by 2002:a05:6808:1190:b0:3bd:9c61:7d41 with SMTP id j16-20020a056808119000b003bd9c617d41mr982809oil.12.1705587765526; Thu, 18 Jan 2024 06:22:45 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-80-239.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.80.239]) by smtp.gmail.com with ESMTPSA id t8-20020a05680800c800b003bd3c878a99sm261380oic.13.2024.01.18.06.22.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 06:22:44 -0800 (PST) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1rQTI3-0051dq-ET; Thu, 18 Jan 2024 10:22:43 -0400 Date: Thu, 18 Jan 2024 10:22:43 -0400 From: Jason Gunthorpe To: "zhangzekun (A)" Cc: will@kernel.org, robin.murphy@arm.com, joro@8bytes.org, nicolinc@nvidia.com, mshavit@google.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH] iommu/arm-smmu-v3: Add a threshold to avoid potential soft lockup Message-ID: <20240118142243.GI50608@ziepe.ca> References: <20240115114040.6279-1-zhangzekun11@huawei.com> <20240115153152.GA50608@ziepe.ca> Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Thu, Jan 18, 2024 at 07:21:10PM +0800, zhangzekun (A) wrote: > > What is the rest of the call chain? How did you get into such a large > > invalidation? > we are doing some test with qemu virtual machines. The testing platform has > 96 cores and 960GB memories. It will enable 64 virtual machines and every > virtual machine will cost 8G memories. When batching closing these 64 > virtaul machine concurrently, we will get soft lockup. > >  WARN: soft lockup - CPU#18 stuck for 11s! [worker:3219] > .... > Call trace: >  dump_backtrace+0x0/0x200 >  show_stack+0x20/0x30 >  dump_stack+0xf0/0x138 >  watchdog_print_info+0x48/0x54 >  watchdog_process_before_softlockup+0x9c/0xa0 >  watchdog_timer_fn+0x1ac/0x2f0 >  __run_hrtimer+0x98/0x2b4 >  __hrtimer_run_queues+0xc0/0x13c >  hrtimer_interrupt+0x150/0x3e4 >  arch_timer_handler_phys+0x3c/0x50 >  handle_percpu_devid_irq+0x90/0x1f4 >  __handle_domain_irq+0x84/0xfc >  gic_handle_irq+0x88/0x2b0 >  el1_irq+0xb8/0x140 >  ktime_get+0x3c/0xac >  arm_smmu_cmdq_poll_until_not_full+0xb0/0x1a0 >  arm_smmu_cmdq_issue_cmdlist+0x194/0x5f4 >  __arm_smmu_tlb_inv_range+0x114/0x22c >  arm_smmu_tlb_inv_walk+0x88/0x120 >  __arm_lpae_unmap+0x188/0x2c0 >  __arm_lpae_unmap+0x104/0x2c0 >  arm_lpae_unmap+0x68/0x80 >  arm_smmu_unmap+0x24/0x40 >  __iommu_unmap+0xd8/0x210 >  iommu_unmap_fast+0x18/0x24 >  unmap_unpin_fast+0x7c/0x140 [vfio_iommu_type1] >  vfio_unmap_unpin+0x14c/0x3ac [vfio_iommu_type1] >  vfio_remove_dma+0x38/0x124 [vfio_iommu_type1] >  vfio_iommu_type1_detach_group+0x4b8/0x4e0 [vfio_iommu_type1] >  __vfio_group_unset_container+0x58/0x18c [vfio] >  vfio_group_try_dissolve_container+0x80/0x94 [vfio] >  vfio_group_put_external_user+0x20/0x54 [vfio] >  kvm_vfio_destroy+0xa8/0x12c >  kvm_destroy_vm+0x20c/0x300 >  kvm_put_kvm+0x74/0xa0 >  kvm_vcpu_release+0x20/0x30 >  __fput+0xc4/0x270 >  ____fput+0x18/0x24 >  task_work_run+0xd0/0x1a0 >  do_exit+0x1d8/0x480 >  do_group_exit+0x40/0x130 >  get_signal+0x1f8/0x744 >  do_signal+0x98/0x204 >  do_notify_resume+0x15c/0x1e0 >  work_pending+0xc/0xa4 I see, that is interesting. I think iommufd is close to substantialy improving this. It already has the domain destruction ordering: - Remove all domain attachments - Read back phys addrs, unmap and unpin - iommu_domain_free() Currently smmu continues to issue ASID invalidations for the domain even though there no references to it. However this is pretty pointless here as we are going to free the ASID right away. I'm going to make a series to allow smmu to support multi-instances on a single domain. In this case when the domain looses all its attachments it looses all its instances too and it won't have to do *anything* for invalidation in this workflow. Ie your flow will boil down to a single ASID invalidation once the domain attachments are all removed, then on invalidations during unmap and no invaludation at free. This would be a significantly faster teardown I suspect. > Thanks for your patch, the patch described below make sense to me. After > spliting TLB invalidate operations according to domain stage, > arm_smmu_tlb_inv_*_s1() can be used both in SVA codes and original smmu > codes, and can implement arm_smmu_tlb_inv_all_s1() just using > arm_smmu_tlb_inv_range_s1() by passing a special size. But the current > smmu-v3 driver has not move "asid" into "struct smmu_domain", we still need > to pass the exact asid from SVA. Yes, it is not intended to be applied on the current kernel, there are a lot of structural problems with how SVA works there.. Jason