From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69EAD1C3F26 for ; Tue, 1 Oct 2024 13:32:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727789564; cv=none; b=PG7jBcPRwWVrjzaZ94cdpmMz9cty3e3Bz9jMZq09QdQl/E3YPQ+HjGDoUlgbcKZ/W6bQfQdZfK2pnI4RXlvtN0t/mX0VYEhMbqh/HZO7/ZXozZowjt69Ln4HPNtJTPfPVD6Xp0G3q97Nxprai5AIs4gBhzOZHryHpgCjsevpBnI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727789564; c=relaxed/simple; bh=PKF8ZVEo8AmNwMLcEwYXeCfaLkhMxlQpxXFFSPA6ZXo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=SmI5KV8Pu2jCS+XMU5KIJl+JtIFGlzqRDXOgq4EoBD4G/Yh627x3GYqL6Xue/zN/Ejj54gg7B1U1TJbDQv2p9OLjsYKwCAAKnoJEEn1oFykdKNuaBAVEO5+allj5NlMRs0CculDmewCX+C2D2CCRFEKO21uO6bWIAKrCtEKhnZk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GSSiOv69; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GSSiOv69" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1727789561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iojT1xfY/iXMYZ4HclXjt5wJyv9eCNn2RMRvon4kbuU=; b=GSSiOv69fNEJVwQ+K86InxXSXuNnKs2jBvWLFV+Ti5gJlurZKFYaUQ/SXfKUgL9gtpi0tx SOo9d1m91pGGgF+6dOwfB9KNX/TWKKkrA6DHWIxMiKGSTxjKO8saZ4vYORC8BbP7kSyIuX e9SbSCkY6I6j8CTCPMrZvnr4ipbfyM4= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-601-83jhRMU3N3iFppyefzT7qg-1; Tue, 01 Oct 2024 09:32:40 -0400 X-MC-Unique: 83jhRMU3N3iFppyefzT7qg-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-42caca7215dso31498745e9.2 for ; Tue, 01 Oct 2024 06:32:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727789559; x=1728394359; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iojT1xfY/iXMYZ4HclXjt5wJyv9eCNn2RMRvon4kbuU=; b=aavAhDwqr0gKt/RFACFBOAOAIi2aza2pT5XyJC04TUzcmwJm7vsn+TUyKaty8HGXSz YL2NUNqjzRhluaeWs14DTXzg4IFdaPRtn6DwOlQ8KgOIV1W/QPJ/HWyzVFKzYVExZ1j2 PmtOPIC0ZHLLghrBQK55SMsCtzFK+lKl5R+5qSqZZtVdsSpj3mfoDYKqU9bZXiNhUtUm dGDkWXudqMULTyOZ2Vefx5bxTeyi6f8vOVCydPw6xWVZwvH4xuC5HjbLMyfeqLpfwlCE 0oqcbWozSTmTKQ+aweZZp4T1DZopZIOWxjF94heaSWcbUXcEt8pXhl+GI3KmTHg3EmTY 2TDw== X-Forwarded-Encrypted: i=1; AJvYcCXdonTyoNLNYRIeWfZFQ4XImO8iuIMNbvxQV5XkMsaFBeZX2fsR0S054Zwp7O0XWwIB8B8=@vger.kernel.org X-Gm-Message-State: AOJu0YxbspA/qmjAevT5nigX8LlUOVtyGRtS1+EAvXlGqATbkM98z9PE kBuVwNwEbbrg3UiLV05TmyqWMa3TwChcw/ztBQ95bj+w3w40ETlxhntpD5OhA5/cnIS5ZH9smmN Ko0W7ImF0Z1yhf8lC3EYHX8sVFcEKOPsoXpg+fnjSej/CEubAUw== X-Received: by 2002:a05:600c:a02:b0:426:66e9:b844 with SMTP id 5b1f17b1804b1-42f584339aemr130160625e9.8.1727789559123; Tue, 01 Oct 2024 06:32:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGDiutjjZjFmvjKmZCcGrKni3t0abnQlYsETvEbcK9G6gugyRexKg3WBxo9lrKH5Dul5fJhqA== X-Received: by 2002:a05:600c:a02:b0:426:66e9:b844 with SMTP id 5b1f17b1804b1-42f584339aemr130159935e9.8.1727789558576; Tue, 01 Oct 2024 06:32:38 -0700 (PDT) Received: from ?IPV6:2a0d:3341:b088:b810::f71? ([2a0d:3341:b088:b810::f71]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42f57e13a1dsm136324005e9.32.2024.10.01.06.32.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 01 Oct 2024 06:32:37 -0700 (PDT) Message-ID: <4968c2ec-5584-4a98-9782-143605117315@redhat.com> Date: Tue, 1 Oct 2024 15:32:34 +0200 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH net v2 2/2] page_pool: fix IOMMU crash when driver has already unbound To: Yunsheng Lin , davem@davemloft.net, kuba@kernel.org Cc: liuyonglong@huawei.com, fanghaiqing@huawei.com, zhangkun09@huawei.com, Robin Murphy , Alexander Duyck , IOMMU , Wei Fang , Shenwei Wang , Clark Wang , Eric Dumazet , Tony Nguyen , Przemek Kitszel , Alexander Lobakin , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Felix Fietkau , Lorenzo Bianconi , Ryder Lee , Shayne Chen , Sean Wang , Kalle Valo , Matthias Brugger , AngeloGioacchino Del Regno , Andrew Morton , Ilias Apalodimas , imx@lists.linux.dev, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org, linux-rdma@vger.kernel.org, linux-wireless@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-mm@kvack.org References: <20240925075707.3970187-1-linyunsheng@huawei.com> <20240925075707.3970187-3-linyunsheng@huawei.com> Content-Language: en-US From: Paolo Abeni In-Reply-To: <20240925075707.3970187-3-linyunsheng@huawei.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 9/25/24 09:57, Yunsheng Lin wrote: > Networking driver with page_pool support may hand over page > still with dma mapping to network stack and try to reuse that > page after network stack is done with it and passes it back > to page_pool to avoid the penalty of dma mapping/unmapping. > With all the caching in the network stack, some pages may be > held in the network stack without returning to the page_pool > soon enough, and with VF disable causing the driver unbound, > the page_pool does not stop the driver from doing it's > unbounding work, instead page_pool uses workqueue to check > if there is some pages coming back from the network stack > periodically, if there is any, it will do the dma unmmapping > related cleanup work. > > As mentioned in [1], attempting DMA unmaps after the driver > has already unbound may leak resources or at worst corrupt > memory. Fundamentally, the page pool code cannot allow DMA > mappings to outlive the driver they belong to. > > Currently it seems there are at least two cases that the page > is not released fast enough causing dma unmmapping done after > driver has already unbound: > 1. ipv4 packet defragmentation timeout: this seems to cause > delay up to 30 secs. > 2. skb_defer_free_flush(): this may cause infinite delay if > there is no triggering for net_rx_action(). > > In order not to do the dma unmmapping after driver has already > unbound and stall the unloading of the networking driver, add > the pool->items array to record all the pages including the ones > which are handed over to network stack, so the page_pool can > do the dma unmmapping for those pages when page_pool_destroy() > is called. As the pool->items need to be large enough to avoid > performance degradation, add a 'item_full' stat to indicate the > allocation failure due to unavailability of pool->items. This looks really invasive, with room for potentially large performance regressions or worse. At very least it does not look suitable for net. Is the problem only tied to VFs drivers? It's a pity all the page_pool users will have to pay a bill for it... /P From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0419CF64BE for ; Tue, 1 Oct 2024 13:32:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 8FEF1400ED; Tue, 1 Oct 2024 13:32:46 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id GZHlEXUYXlmD; Tue, 1 Oct 2024 13:32:45 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 7F277400D2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1727789565; bh=iojT1xfY/iXMYZ4HclXjt5wJyv9eCNn2RMRvon4kbuU=; h=Date:To:References:From:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=NtekzxBx0M5SEys6WTIEoVWsrrbFQPsqy63mkcPtM51Ci4Tv6pxn3ID7Hm0jcfmae kGiduWmE4gNua+U0k5/mUc/kUu9QiDteBp6/p6Qy6qJOPt7z/y2fxDK7bgdfrmgFNH SYyilp2HI3G/YgPF4xVzV6soWEZY8+ns6CbsXkfceCsLL49u0Gemz/4PVXOkmA9+n3 NciDq+CTk94gbAwr1VJqDEzK4VcQbklOkTcYDfSYimiqtjqJafzgXfImonc6ub+bSb s+0uoMrxpzlh2QEEJVI220H2fOhJurceCkG0VDpIS13X2/en+3A64PEI3aWWIwSRub PgNOxhao4oVkw== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 7F277400D2; Tue, 1 Oct 2024 13:32:45 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 057B61BF3DE for ; Tue, 1 Oct 2024 13:32:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 006E560839 for ; Tue, 1 Oct 2024 13:32:44 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id gNxOR8Z5045B for ; Tue, 1 Oct 2024 13:32:43 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=pabeni@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 1C351607B4 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 1C351607B4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 1C351607B4 for ; Tue, 1 Oct 2024 13:32:42 +0000 (UTC) Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-335-E8QkU2agPE2QRRmnQSJ3qw-1; Tue, 01 Oct 2024 09:32:40 -0400 X-MC-Unique: E8QkU2agPE2QRRmnQSJ3qw-1 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-42cb89fbb8cso31589315e9.0 for ; Tue, 01 Oct 2024 06:32:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727789559; x=1728394359; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iojT1xfY/iXMYZ4HclXjt5wJyv9eCNn2RMRvon4kbuU=; b=NJ+oHtTTS29W6nCw9IBD5JuKEalFHKs84eFM2i+JvRzR9ZQsQlVM+gmUzPsWCg/8/d c5mUsFNGzqyaEFOUax4pbIY8SrlIyE62UQaEuAzc7i1gkiMyRxcPoSpNCtuQBQeZAXE7 CCm/ZKNWEGJlGtwZbO/unLWuFdk5I/RvyHeKGVjrP9Y9e4Q9HNahwRH+iGi2LgEq9xnP yNhRLy64y8RRdgejVaNkqvsQmg+7qCXFAa84oASQdsWYU3W2u+vsF59X+bCBiWub4ctp xcleqzSf0TJMK2CuV3P10cmweoXPRhZYFkUDO2i6uNKVA3n3hRKZokmwZrWNPhIoOcAG A1GQ== X-Forwarded-Encrypted: i=1; AJvYcCUyIzVLPjooQ4WLJLQIzeWGj+SoA1VbCDluG6YecqCE7fGEXYbN6L5ZmEXyiM0rsDaFo0NXl1PDcVbf+bAgbEk=@lists.osuosl.org X-Gm-Message-State: AOJu0Yygz1R83Bz0qHOlCF0JLRTR/8alox/hVfKm+QQFoLCNgSxR7Oay g+V96T530azzgWEbZ73/fSilqVsp22SxuXp867+4GdVeRL5uEtRUR2VVGyl/B0M0sHegNH+afTv qub0KPBtwpxAtbFBD98IM1iOQwmhMNRkwBm+S1IqRYeM//34W7X4oOFGy6SLp2hfRLcY= X-Received: by 2002:a05:600c:a02:b0:426:66e9:b844 with SMTP id 5b1f17b1804b1-42f584339aemr130160525e9.8.1727789559104; Tue, 01 Oct 2024 06:32:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGDiutjjZjFmvjKmZCcGrKni3t0abnQlYsETvEbcK9G6gugyRexKg3WBxo9lrKH5Dul5fJhqA== X-Received: by 2002:a05:600c:a02:b0:426:66e9:b844 with SMTP id 5b1f17b1804b1-42f584339aemr130159935e9.8.1727789558576; Tue, 01 Oct 2024 06:32:38 -0700 (PDT) Received: from ?IPV6:2a0d:3341:b088:b810::f71? ([2a0d:3341:b088:b810::f71]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42f57e13a1dsm136324005e9.32.2024.10.01.06.32.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 01 Oct 2024 06:32:37 -0700 (PDT) Message-ID: <4968c2ec-5584-4a98-9782-143605117315@redhat.com> Date: Tue, 1 Oct 2024 15:32:34 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Yunsheng Lin , davem@davemloft.net, kuba@kernel.org References: <20240925075707.3970187-1-linyunsheng@huawei.com> <20240925075707.3970187-3-linyunsheng@huawei.com> From: Paolo Abeni In-Reply-To: <20240925075707.3970187-3-linyunsheng@huawei.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1727789561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iojT1xfY/iXMYZ4HclXjt5wJyv9eCNn2RMRvon4kbuU=; b=GSSiOv69fNEJVwQ+K86InxXSXuNnKs2jBvWLFV+Ti5gJlurZKFYaUQ/SXfKUgL9gtpi0tx SOo9d1m91pGGgF+6dOwfB9KNX/TWKKkrA6DHWIxMiKGSTxjKO8saZ4vYORC8BbP7kSyIuX e9SbSCkY6I6j8CTCPMrZvnr4ipbfyM4= X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=GSSiOv69 Subject: Re: [Intel-wired-lan] [PATCH net v2 2/2] page_pool: fix IOMMU crash when driver has already unbound X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: imx@lists.linux.dev, Alexei Starovoitov , Alexander Duyck , linux-mm@kvack.org, Eric Dumazet , Tony Nguyen , Shenwei Wang , Ryder Lee , Daniel Borkmann , linux-rdma@vger.kernel.org, Przemek Kitszel , John Fastabend , IOMMU , liuyonglong@huawei.com, Clark Wang , zhangkun09@huawei.com, fanghaiqing@huawei.com, Lorenzo Bianconi , Jesper Dangaard Brouer , Kalle Valo , Sean Wang , Wei Fang , Matthias Brugger , intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, AngeloGioacchino Del Regno , Leon Romanovsky , Saeed Mahameed , Ilias Apalodimas , linux-wireless@vger.kernel.org, linux-kernel@vger.kernel.org, Tariq Toukan , Alexander Lobakin , netdev@vger.kernel.org, linux-mediatek@lists.infradead.org, Andrew Morton , Shayne Chen , Robin Murphy , Felix Fietkau Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" On 9/25/24 09:57, Yunsheng Lin wrote: > Networking driver with page_pool support may hand over page > still with dma mapping to network stack and try to reuse that > page after network stack is done with it and passes it back > to page_pool to avoid the penalty of dma mapping/unmapping. > With all the caching in the network stack, some pages may be > held in the network stack without returning to the page_pool > soon enough, and with VF disable causing the driver unbound, > the page_pool does not stop the driver from doing it's > unbounding work, instead page_pool uses workqueue to check > if there is some pages coming back from the network stack > periodically, if there is any, it will do the dma unmmapping > related cleanup work. > > As mentioned in [1], attempting DMA unmaps after the driver > has already unbound may leak resources or at worst corrupt > memory. Fundamentally, the page pool code cannot allow DMA > mappings to outlive the driver they belong to. > > Currently it seems there are at least two cases that the page > is not released fast enough causing dma unmmapping done after > driver has already unbound: > 1. ipv4 packet defragmentation timeout: this seems to cause > delay up to 30 secs. > 2. skb_defer_free_flush(): this may cause infinite delay if > there is no triggering for net_rx_action(). > > In order not to do the dma unmmapping after driver has already > unbound and stall the unloading of the networking driver, add > the pool->items array to record all the pages including the ones > which are handed over to network stack, so the page_pool can > do the dma unmmapping for those pages when page_pool_destroy() > is called. As the pool->items need to be large enough to avoid > performance degradation, add a 'item_full' stat to indicate the > allocation failure due to unavailability of pool->items. This looks really invasive, with room for potentially large performance regressions or worse. At very least it does not look suitable for net. Is the problem only tied to VFs drivers? It's a pity all the page_pool users will have to pay a bill for it... /P