Re: Problem with smbdirect rw credits and initiator_depth

[Stefan Metzmacher <metze@samba.org>]

Am 18.01.26 um 09:03 schrieb Namjae Jeon:
> On Sat, Jan 17, 2026 at 10:15 PM Stefan Metzmacher <metze@samba.org> wrote:
>>
>> Am 17.01.26 um 00:08 schrieb Stefan Metzmacher:
>>> Am 15.01.26 um 10:50 schrieb Stefan Metzmacher:
>>>> Am 15.01.26 um 03:01 schrieb Namjae Jeon:
>>>>> On Thu, Jan 15, 2026 at 3:13 AM Stefan Metzmacher <metze@samba.org> wrote:
>>>>>>
>>>>>> Am 15.12.25 um 21:17 schrieb Stefan Metzmacher:
>>>>>>> Am 14.12.25 um 23:56 schrieb Stefan Metzmacher:
>>>>>>>> Am 13.12.25 um 03:14 schrieb Namjae Jeon:
>>>>>>>>>> I've put these changes a long with rw credit fixes into my
>>>>>>>>>> for-6.18/ksmbd-smbdirect-regression-v4 branch, are you able to
>>>>>>>>>> test this?
>>>>>>>>> Problems still occur. See:
>>>>>>>>
>>>>>>>> :-( Would you be able to use rxe and cake a network capture?
>>>>>>>>
>>>>>>>> Using test files with all zeros, e.g.
>>>>>>>> dd if=/dev/zero of=/tmp/4096MBzeros-sparse.dat seek=4096MB bs=1 count=1
>>>>>>>> would allow gzip --best on the capture file to compress well...
>>>>>>>
>>>>>>> I think I found something that explains it and
>>>>>>> I was able to reproduce and what I have in mind.
>>>>>>>
>>>>>>> We increment recv_io.posted.count after ib_post_recv()
>>>>>>>
>>>>>>> And manage_credits_prior_sending() uses
>>>>>>>
>>>>>>> new_credits = recv_io.posted.count - recv_io.credits.count
>>>>>>>
>>>>>>> But there is a race between the hardware receiving a message
>>>>>>> and recv_done being called in order to decrement recv_io.posted.count
>>>>>>> again. During that race manage_credits_prior_sending() might grant
>>>>>>> too much credits.
>>>>>>>
>>>>>>> Please test my for-6.18/ksmbd-smbdirect-regression-v5 branch,
>>>>>>> I haven't tested this branch yet, I'm running out of time
>>>>>>> for the day.
>>>>>>>
>>>>>>> But I tested it with smbclient and having a similar
>>>>>>> logic in fs/smb/common/smbdirect/smbdirect_connection.c
>>>>>>
>>>>>> I was able to reproduce the problem and the fix I created
>>>>>> for-6.18/ksmbd-smbdirect-regression-v5 was not correct.
>>>>>>
>>>>>> I needed to use
>>>>>>
>>>>>> available = atomic_xchg(&sc->recv_io.credits.available, 0);
>>>>>>
>>>>>> instead of
>>>>>>
>>>>>> available = atomic_read(&sc->recv_io.credits.available);
>>>>>> atomic_sub(new_credits, &sc->recv_io.credits.available);
>>>>>>
>>>>>> This following branch works for me:
>>>>>> for-6.18/ksmbd-smbdirect-regression-v7
>>>>>> and with the fixes again master this should also work:
>>>>>> for-6.19/ksmbd-smbdirect-regression-v1
>>>>>>
>>>>>> I'll post real patches tomorrow.
>>>>>>
>>>>>> Please check.
>>>>> Okay, I will test it with two branches.
>>>>> I'll try it too, but I recommend running frametest for performance
>>>>> difference and stress testing.
>>>>>
>>>>> https://support.dvsus.com/hc/en-us/articles/212925466-How-to-use-frametest
>>>>>
>>>>> ex) frametest.exe -w 4k -t 20 -n 2000
>>>>
>>>> That works fine, but
>>>>
>>>>    frametest.exe -r 4k -t 20 -n 2000
>>>>
>>>> generates a continues stream of such messages:
>>>> ksmbd: Failed to send message: -107
>>>>
>>>> Both with 6.17.2 and for-6.19/ksmbd-smbdirect-regression-v1,
>>>> so this is not a regression.
>>>>
>>>> I'll now check if the is related to the other problems
>>>> I found and fixes in for-6.18/ksmbd-smbdirect-regression-v5
>>>
>>> Ok, I found the problem.
>>>
>>> On send we are not allowed to consume the last send credit
>>> without granting any credit to the peer.
>>>
>>>       MS-SMBD 3.1.5.1 Sending Upper Layer Messages
>>>
>>>       ...
>>>       If Connection.SendCredits is 1 and the CreditsGranted field of the message is 0, stop
>>>       processing.
>>>       ...
>>>
>>>       MS-SMBD 3.1.5.9 Managing Credits Prior to Sending
>>>
>>>       ...
>>>       If Connection.ReceiveCredits is zero, or if Connection.SendCredits is one and the
>>>       Connection.SendQueue is not empty, the sender MUST allocate and post at least one receive of size
>>>       Connection.MaxReceiveSize and MUST increment Connection.ReceiveCredits by the number
>>>       allocated and posted. If no receives are posted, the processing MUST return a value of zero to indicate
>>>       to the caller that no Send message can be currently performed.
>>>       ...
>>>
>>> It works in my master-ipproto-smbdirect branch, see the top commit.
>>>
>>> I'll backport the related logic to ksmbd on top of
>>> for-6.19/ksmbd-smbdirect-regression-v1 tomorrow.
>>
>> for-6.19/ksmbd-smbdirect-regression-v2 has the fixes and works for
>> me, I'll prepare official patches (most likely) on Monday.
> I have tested the for-6.19/ksmbd-smbdirect-regression-v2 branch, and I
> can confirm that the issues I previously encountered in my test
> environment have been fixed.

Great! Thanks for testing!

> I have a couple of follow-up questions regarding this fix:
> 1. Regarding your frametest results, did you not observe any
> performance degradation or difference compared to linux-6.17.9?

Sorry, I don't understand what you are asking for.

Do you mean with v6.19-rc5, for-6.19/ksmbd-smbdirect-regression-v1 or
for-6.19/ksmbd-smbdirect-regression-v2?


> 2. You mentioned previously testing with Intel E810-CQDA2 NICs. Have
> you tested both iWARP and RoCEv2 modes on the E810?

Yes, both while there seem to be strange problems with iWarp.

I'll have to re-test with these cards, we'll test if it's possible
to have both cards installed together both only getting 8 PCIe 5 lanes,
that would make it easier to test.

At the time I was always testing with KSAN, lockdep and other debugging features
turned on, so performance was not as expected anyway...

metze