Re: [PATCH net-next v3 1/2] net/smc: make wr buffer count configurable

[Dust Li <dust.li@linux.alibaba.com>]

On 2025-09-28 10:39:51, Halil Pasic wrote:
>On Sun, 28 Sep 2025 10:02:43 +0800
>Dust Li <dust.li@linux.alibaba.com> wrote:
>
>> >Unfortunately I don't quite understand why qp_attr.cap.max_send_wr is 3
>> >times the number of send WR buffers we allocate. My understanding
>> >is that qp_attr.cap.max_send_wr is about the number of send WQEs.  
>> 
>> We have at most 2 RDMA Write for 1 RDMA send. So 3 times is necessary.
>> That is explained in the original comments. Maybe it's better to keep it.
>> 
>> ```
>> .cap = {
>>                 /* include unsolicited rdma_writes as well,
>>                  * there are max. 2 RDMA_WRITE per 1 WR_SEND
>>                  */
>
>But what are "the unsolicited" rdma_writes? I have heard of
>unsolicited receive, where the data is received without
>consuming a WR previously put on the RQ on the receiving end, but
>the concept of unsolicited rdma_writes eludes me completely.

unsolicited RDMA Writes means those RDMA Writes won't generate
CQEs on the local side. You can refer to:
https://www.rdmamojo.com/2014/05/27/solicited-event/

>
>I guess what you are trying to say, and what I understand is
>that we first put the payload into the RMB of the remote, which
>may require up 2 RDMA_WRITE operations, probably because we may
>cross the end (and start) of the array that hosts the circular
>buffer, and then we send a CDC message to update the cursor.
>
>For the latter a  ib_post_send() is used in smc_wr_tx_send()
>and AFAICT it consumes a WR from wr_tx_bufs. For the former
>we consume a single wr_tx_rdmas which and each wr_tx_rdmas
>has 2 WR allocated.

Right.

>
>And all those WRs need a WQE. So I guess now I do understand
>SMC_WR_BUF_CNT, but I find the comment still confusing like
>hell because of these unsolicited rdma_writes.
>
>Thank you for the explanation! It was indeed helpful! Let
>me try to come up with a better comment -- unless somebody
>manages to explain "unsolicited rdma_writes" to me.
>
>>         .max_send_wr = SMC_WR_BUF_CNT * 3,
>>         .max_recv_wr = SMC_WR_BUF_CNT * 3,
>>         .max_send_sge = SMC_IB_MAX_SEND_SGE,
>>         .max_recv_sge = lnk->wr_rx_sge_cnt,
>>         .max_inline_data = 0,
>> },
>> ```
>> 
>> >I assume that qp_attr.cap.max_send_wr == qp_attr.cap.max_recv_wr
>> >is not something we would want to preserve.  
>> 
>> IIUC, RDMA Write won't consume any RX wqe on the receive side, so I think
>> the .max_recv_wr can be SMC_WR_BUF_CNT if we don't use RDMA_WRITE_IMM.
>
>Maybe we don't want to assume somebody else (another implementation)
>would not use immediate data. I'm not sure. But I don't quite understand
>the why the relationship between the send and the receive side either.

I missed something here. I sent an other email right after this to
explain my thoughts here:

    I kept thinking about this a bit more, and I realized that max_recv_wr
    should be larger than SMC_WR_BUF_CNT.

    Since receive WQEs are posted in a softirq context, their posting may be
    delayed. Meanwhile, the sender might already have received the TX
    completion (CQE) and continue sending new messages. In this case, if the
    receiver’s post_recv() (i.e., posting of RX WQEs) is delayed, an RNR
    (Receiver Not Ready) can easily occur.

Best regards,
Dust