From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-181.mta0.migadu.com (out-181.mta0.migadu.com [91.218.175.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3B4E14EC79 for ; Tue, 7 May 2024 15:24:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715095498; cv=none; b=OvBEJkWD9zu7eE2+uLmsjcLMJDej04JMuKvoy2lSRknsnmiHtcd+ohIrOIyJURAcbNfH6BO1Cp1A8ZBa0HmBa3KjIGqcrI6LDqHBwnO8h6ZQa6MzlCtdTKqWVuryVTkeXE2rE9cqlLWsn4f7I/RWFtzoXCyUkzQeistq6I3Yc9g= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715095498; c=relaxed/simple; bh=wqk9kva6Z468fnzR4dhmllrDiCYNXiexDSOPYNO459Q=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=jfzUE9jp1yJGbvXGTNr/UJELl2wz6YZdmcDtkBZS8LkERkfw1tnh2ftThphAkmbZWfgBN07MYh493abClhIjbYWGlLgAUUCMsCM/q13VHYgGh7ABZhodm0eL9ZkaQw1RJE+QAQC5CPp553gQn8KeWzZDBJInAZDhamMPR6yydSo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=vXpt7VoD; arc=none smtp.client-ip=91.218.175.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="vXpt7VoD" Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1715095495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5IpRWdMlVP/9sg2sRxny0HoNTCrMVdhtZN7f2dJNDTE=; b=vXpt7VoDsziyTOU19E1lKXsYOxJXnsOT90C0F4jXgonZfGkJybH9/+FLAJ9C3OikcF3Sag sT3LTwRQc5/sMbhHXEMUOa9jKaJmGtVzL9WiXjJRL3EjZA1p6H0t0hNkRQ8YbJdN8bj506 e1QUk69/JcQsrfsTB0sab8b3PGBCQtY= Date: Tue, 7 May 2024 17:24:51 +0200 Precedence: bulk X-Mailing-List: stable@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Subject: Re: Excessive memory usage when infiniband config is enabled To: Konstantin Taranov , Brian Baboch , Leon Romanovsky Cc: "linux-rdma@vger.kernel.org" , "stable@vger.kernel.org" , "florent.fourcot@wifirst.fr" , "brian.baboch@wifirst.fr" References: <2b4f7f6e-9fe5-43a4-b62e-6e42be67d1d9@gmail.com> <20240507112730.GB78961@unreal> <8992c811-f8d9-4c95-8931-ee4a836d757e@gmail.com> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Zhu Yanjun In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT 在 2024/5/7 15:32, Konstantin Taranov 写道: >> Hello Leon, >> >> I feel that it's a bug because I don't understand why is this module/option >> allocating 6GB of RAM without any explicit configuration or usage from us. >> It's also worth mentioning that we are using the default linux-image from >> Debian bookworm, and it took us a long time to understand the reason >> behind this memory increase by bisecting the kernel's config file. >> Moreover the documentation of the module doesn't mention anything >> regarding additional memory usage, we're talking about an increase of 6Gb >> which is huge since we're not using the option. >> So is that an expected behavior, to have this much increase in the memory >> consumption, when activating the RDMA option even if we're not using it ? If >> that's the case, perhaps it would be good to mention this in the >> documentation. >> >> Thank you >> > > Hi Brian, > > I do not think it is a bug. The high memory usage seems to come from these lines: > rsrc_size = irdma_calc_mem_rsrc_size(rf); > rf->mem_rsrc = vzalloc(rsrc_size); Exactly. The memory usage is related with the number of QP. When on irdma, the Queue Pairs is 4092, Completion Queues is 8189, the memory usage is about 4194302. The command "modprobe irdma limits_sel" will change QP numbers. 0 means minimum, up to 124 QPs. Please use the command "modprobe irdma limits_sel=0" to make tests. Please let us know the test results. Zhu Yanjun > > inside of irdma_initialize_hw_rsrc function. You can read the code of > irdma_calc_mem_rsrc_size to understand the 6GB memory usage. > > You can ask developers of irdma to optimize memory usage. > Btw., module is loaded == module is used. There is no "loaded and unused". > > Konstantin