Re: Re: nvme-tcp: fix a possible UAF when failing to send request

["zhang.guanghui@cestc.cn" <zhang.guanghui@cestc.cn>]

Hi 
        After testing this patch,  No sending request failure occurred, the issue has not been reproduced yet.  
It may take a long time to test.

best wishes



zhang.guanghui@cestc.cn



 



发件人： Sagi Grimberg



发送时间： 2025-02-17 15:46



收件人： zhang.guanghui@cestc.cn; mgurtovoy; kbusch; sashal; chunguang.xu



抄送： linux-kernel; linux-nvme; linux-block



主题： Re: nvme-tcp: fix a possible UAF when failing to send request【请注意，邮件由sagigrim@gmail.com代发】



 



 



 



On 10/02/2025 9:41, zhang.guanghui@cestc.cn wrote:



> Hello



>



>



>



>      When using the nvme-tcp driver in a storage cluster, the driver may trigger a null pointer causing the host to crash several times.



>



>



>



> By analyzing the vmcore, we know the direct cause is that  the request->mq_hctx was used after free.



>



>



>



>



>



> CPU1                                                                   CPU2



>



>



>



> nvme_tcp_poll                                                          nvme_tcp_try_send  --failed to send reqrest 13



>



>



>



>      nvme_tcp_try_recv                                                      nvme_tcp_fail_request



>



>



>



>          nvme_tcp_recv_skb                                                      nvme_tcp_end_request



>



>



>



>              nvme_tcp_recv_pdu                                                      nvme_complete_rq



>



>



>



>                  nvme_tcp_handle_comp                                                   nvme_retry_req -- request->mq_hctx have been freed, is NULL.



>



>



>



>                      nvme_tcp_process_nvme_cqe



>



>



>



>                          nvme_complete_rq



>



>



>



>                              nvme_end_req



>



>



>



>                                    blk_mq_end_request



>



>



>



>



>



>



>



> when nvme_tcp_try_send failed to send reqrest 13, it maybe be resulted by selinux or other reasons, this is a problem. then  the nvme_tcp_fail_request would execute。



>



>



>



> but the nvme_tcp_recv_pdu may have received the responding pdu and the nvme_tcp_process_nvme_cqe would have completed the request.  request->mq_hctx was used after free.



>



>



>



> the follow patch is to solve it.



 



Zhang, your email client needs fixing - it is impossible to follow your



emails.



 



>



>



>



> can you give  some suggestions?  thanks!



 



The problem is the C2HTerm that the host is unable to handle correctly.



And it also appears that nvme_tcp_poll() does not signal correctly to



blk-mq to stop



calling poll.



 



One thing to do is to handle C2HTerm PDU correctly, and, here is a



possible fix to try for the UAF issue:



--



diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c



index c637ff04a052..0e390e98aaf9 100644



--- a/drivers/nvme/host/tcp.c



+++ b/drivers/nvme/host/tcp.c



@@ -2673,6 +2673,7 @@ static int nvme_tcp_poll(struct blk_mq_hw_ctx



*hctx, struct io_comp_batch *iob)



 {



        struct nvme_tcp_queue *queue = hctx->driver_data;



        struct sock *sk = queue->sock->sk;



+       int ret;



 



        if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))



                return 0;



@@ -2680,9 +2681,9 @@ static int nvme_tcp_poll(struct blk_mq_hw_ctx



*hctx, struct io_comp_batch *iob)



        set_bit(NVME_TCP_Q_POLLING, &queue->flags);



        if (sk_can_busy_loop(sk) &&



skb_queue_empty_lockless(&sk->sk_receive_queue))



                sk_busy_loop(sk, true);



-       nvme_tcp_try_recv(queue);



+       ret = nvme_tcp_try_recv(queue);



        clear_bit(NVME_TCP_Q_POLLING, &queue->flags);



-       return queue->nr_cqe;



+       return ret < 0 ? ret : queue->nr_cqe;



 }



 



 static int nvme_tcp_get_address(struct nvme_ctrl *ctrl, char *buf, int



size)



--



 



Does this help?