From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from hr2.samba.org (hr2.samba.org [144.76.82.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E6C32EB846; Wed, 10 Dec 2025 16:42:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.76.82.148 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765384943; cv=none; b=Dui+qiCsOjWlFA9FgKi/ih/RU2rjcTWl5FqTpLO7nJtrJ5swK/s6omCro4UjjSe6sCrRFsIF79PJyI376vhMRGxV5G2smDXDUTQM2TPawy0POLwSUER5XfOr5B0ipA0YinMQwi6mR2dUW1y7a8KcaNZGQH6GHb7I3J4hzxLxGkY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765384943; c=relaxed/simple; bh=l2Ld7sBVKu8YDuGF611lQ99Ji+wx0fzq1oAUp+3Zin4=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=qCwYVDchUBJW1kCj2+KITcGV3m/UZWbs9LPwLDwR/r5AUdQv/d8QN83PRtHLIi+5ZJ4bNSqUe9qK15KzqVMhXozz4O5slNPLJIEd5BFhnA4Pv23PTorb0t0PBQiahICaLuwYZiy4W7B+WNEMqFoZnaZTOm/Z95a1A4nFkcAeHN4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=samba.org; spf=pass smtp.mailfrom=samba.org; dkim=pass (3072-bit key) header.d=samba.org header.i=@samba.org header.b=Hcgn5/hR; arc=none smtp.client-ip=144.76.82.148 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=samba.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samba.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (3072-bit key) header.d=samba.org header.i=@samba.org header.b="Hcgn5/hR" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=From:Cc:To:Date:Message-ID; bh=26mRdyj5ukbuYOEXWu6R6f0fK68V9mMcyWoL9cL1sjs=; b=Hcgn5/hRKy90FDEhzarkrU/qLW d3VFjGJ4wWKwq9j6I4E9zRDxRVOfuUckQoR9SaThmrNZ0R3GfjTEoNhTB+dqONgqG6+X2sxD6Zgie Lk+E29uGzs4NqjgCaBNa1z018SqCshH5dYA1MUXwVB6vAWtQbJJt6c9ats3+TMgtiQbMFS/5mZbin 2H5kuVR4W5tuk2/I5twBCAs2qZLM6EC8kuV+deTNztoSZ7Q8lAnIRshn2UQXa+qAceQUGesyPWAs7 j+sW1nbR+nBKmJn4P/cLH+7c7khW2+2U6Erl6W4EziWGEH2ZjSAHHnR/gmpfcue1uRq3NePoOQLRb 4Qf41Y7T76UFyka9W6GJxWvALezybfHr8F9z+vHKqGI8Oc4PK68lwQFhk1Ar17ZzR56Xr+DVkM0O1 6zDDHanHUeQxN8/VjHS7P9xkrZzoOjERI6W3z5P+Qto96n/Zsm3mRQ8Y8RhCTQ+Ay47XcX/lS+Ltb 2BceQJjNVzlAh7I7AMZ36RHR; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1vTNGW-00086L-2I; Wed, 10 Dec 2025 16:42:12 +0000 Message-ID: <86b3c222-d765-4a6c-bb79-915609fa3d27@samba.org> Date: Wed, 10 Dec 2025 17:42:12 +0100 Precedence: bulk X-Mailing-List: linux-cifs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Problem with smbdirect rw credits and initiator_depth To: Namjae Jeon Cc: Tom Talpey , "linux-cifs@vger.kernel.org" , "linux-rdma@vger.kernel.org" References: <35eec2e6-bf37-43d6-a2d8-7a939a68021b@samba.org> Content-Language: en-US From: Stefan Metzmacher In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Am 05.12.25 um 13:21 schrieb Namjae Jeon: >>> Can you at least post the dmesg output generated by this: >>> https://git.samba.org/?p=metze/linux/wip.git;a=commitdiff;h=7e724ebc58e986f4e101a55f4ab5e96912239918 >>> Assuming that this wasn't triggered: >>> if (WARN_ONCE(needed > max_possible, "needed:%u > max:%u\n", needed, max_possible)) >> I didn't know you wanted it. I will share it after office. > I have attached v2 and v3 logs. Let me know if you need something more, >>> >>> Did you run the bpftrace command? Did it print a lot of >>> 'smb_direct_rdma_xmit' message over the whole time of the file copy? >> No, I didn't check it. but I will try this. > /mnt# bpftrace ksmbd-rdma-xmit.bt > Attaching 1 probe... > > The absence of any output after Attaching 1 probe... indicates that > the smb_direct_rdma_xmit function has not been called ? Assuming the client requires signing, I may found the reason for a recv credit problem. ksmbd uses this: smb_direct_max_fragmented_recv_size = 1024 * 1024 smb_direct_max_receive_size = 1364; smb_direct_receive_credit_max = 255; In order for the client to fill the full eassembly buffer, all our recv buffers are moved into it, which means 255 * (1364 - 24) = 341700 (0x536C4) bytes of payload, after that we no longer able to grant and new recv credits to the peer, which tries to send up to 1048576 (0x100000). I found this using smbclient to download a large file from a Windows server without using rdma offload. So I guess you are seeing the problem when Windows tries to copy a file to ksmbd. For smbclient I made it work by changing max_fragmented_recv_size to the minimum value of 131072 (0x20000), this value is smaller than all local recv buffers 255 * (1364 - 24) = 341700 (0x536C4). I try to find what difference we have between 6.17.9 and 6.18 tomorrow. In the meantime you may want to test if 6.18 with smb_direct_max_fragmented_recv_size = 131072 works for you, or change smb_direct_receive_credit_max = 1024. metze