[PATCH] nvme-tcp: fix memory leak when freeing a queue

[PATCH] nvme-tcp: fix memory leak when freeing a queue

[Maurizio Lombardi]

Release the page frag cache when tearing down the io queues

Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
---
 drivers/nvme/host/tcp.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 07156ea9d1a8..999417626b15 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1229,6 +1229,7 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
 
 static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 {
+	struct page *page;
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
 	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
 
@@ -1238,6 +1239,11 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	if (queue->hdr_digest || queue->data_digest)
 		nvme_tcp_free_crypto(queue);
 
+	if (queue->pf_cache.va) {
+		page = virt_to_head_page(queue->pf_cache.va);
+		__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
+		queue->pf_cache.va = NULL;
+	}
 	sock_release(queue->sock);
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
-- 
2.27.0

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[Sagi Grimberg]

> Release the page frag cache when tearing down the io queues
> 
> Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
> ---
>   drivers/nvme/host/tcp.c | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 07156ea9d1a8..999417626b15 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -1229,6 +1229,7 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
>   
>   static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
>   {
> +	struct page *page;
>   	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
>   	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
>   
> @@ -1238,6 +1239,11 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
>   	if (queue->hdr_digest || queue->data_digest)
>   		nvme_tcp_free_crypto(queue);
>   
> +	if (queue->pf_cache.va) {
> +		page = virt_to_head_page(queue->pf_cache.va);
> +		__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
> +		queue->pf_cache.va = NULL;
> +	}

Thanks Maurizio,

Why do we need to the pf_cache.va protection? nvme_tcp_free_queue
is already protected against re-entry with the NVME_TCP_Q_ALLOCATED
bit.

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[Maurizio Lombardi]

On Wed, Nov 03, 2021 at 11:23:45AM +0200, Sagi Grimberg wrote:
> 
> > Release the page frag cache when tearing down the io queues
> > 
> > Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
> > ---
> >   drivers/nvme/host/tcp.c | 6 ++++++
> >   1 file changed, 6 insertions(+)
> > 
> > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> > index 07156ea9d1a8..999417626b15 100644
> > --- a/drivers/nvme/host/tcp.c
> > +++ b/drivers/nvme/host/tcp.c
> > @@ -1229,6 +1229,7 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
> >   static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
> >   {
> > +	struct page *page;
> >   	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
> >   	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
> > @@ -1238,6 +1239,11 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
> >   	if (queue->hdr_digest || queue->data_digest)
> >   		nvme_tcp_free_crypto(queue);
> > +	if (queue->pf_cache.va) {
> > +		page = virt_to_head_page(queue->pf_cache.va);
> > +		__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
> > +		queue->pf_cache.va = NULL;
> > +	}
> 
> Thanks Maurizio,
> 
> Why do we need to the pf_cache.va protection? nvme_tcp_free_queue
> is already protected against re-entry with the NVME_TCP_Q_ALLOCATED
> bit.
> 

It's not the re-entry what worried me, I thought that nvme_tcp_free_queue()
might be called before page_frag_alloc() had the chance to initialize the pf_cache, triggering
a NULL pointer dereference. I am doing some tests right now and it seems not to be
possible so maybe we can drop the "if".

However, we still need to set pf_cache.va to NULL because the queues might be re-used
in the error code path and we have to prevent page_frag_alloc() from dereferencing a
freed cache page and corrupting the memory.

Maurizio

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[Maurizio Lombardi]

> > 
> > Thanks Maurizio,
> > 
> > Why do we need to the pf_cache.va protection? nvme_tcp_free_queue
> > is already protected against re-entry with the NVME_TCP_Q_ALLOCATED
> > bit.
> > 
> 
> It's not the re-entry what worried me, I thought that nvme_tcp_free_queue()
> might be called before page_frag_alloc() had the chance to initialize the pf_cache, triggering
> a NULL pointer dereference. I am doing some tests right now and it seems not to be
> possible so maybe we can drop the "if".

Oh wait, if nvme_tcp_setup_ctrl() fails it could call nvme_tcp_destroy_io_queues() and
iI guess that in that case the pf_cache is not initialized, so the if(pf_cache.va) protection
is necessary.

Maurizio

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[Christoph Hellwig]

On Wed, Nov 03, 2021 at 12:03:10PM +0100, Maurizio Lombardi wrote:
> > > 
> > > Thanks Maurizio,
> > > 
> > > Why do we need to the pf_cache.va protection? nvme_tcp_free_queue
> > > is already protected against re-entry with the NVME_TCP_Q_ALLOCATED
> > > bit.
> > > 
> > 
> > It's not the re-entry what worried me, I thought that nvme_tcp_free_queue()
> > might be called before page_frag_alloc() had the chance to initialize the pf_cache, triggering
> > a NULL pointer dereference. I am doing some tests right now and it seems not to be
> > possible so maybe we can drop the "if".
> 
> Oh wait, if nvme_tcp_setup_ctrl() fails it could call nvme_tcp_destroy_io_queues() and
> iI guess that in that case the pf_cache is not initialized, so the if(pf_cache.va) protection
> is necessary.

Sagi, does this look good to you now?

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[John Meneghini]

Looks good to me.

Reviewed-by: John Meneghini <jmeneghi@redhat.com>

On 11/9/21 03:13, Christoph Hellwig wrote:
> On Wed, Nov 03, 2021 at 12:03:10PM +0100, Maurizio Lombardi wrote:
>>>>
>>>> Thanks Maurizio,
>>>>
>>>> Why do we need to the pf_cache.va protection? nvme_tcp_free_queue
>>>> is already protected against re-entry with the NVME_TCP_Q_ALLOCATED
>>>> bit.
>>>>
>>>
>>> It's not the re-entry what worried me, I thought that nvme_tcp_free_queue()
>>> might be called before page_frag_alloc() had the chance to initialize the pf_cache, triggering
>>> a NULL pointer dereference. I am doing some tests right now and it seems not to be
>>> possible so maybe we can drop the "if".
>>
>> Oh wait, if nvme_tcp_setup_ctrl() fails it could call nvme_tcp_destroy_io_queues() and
>> iI guess that in that case the pf_cache is not initialized, so the if(pf_cache.va) protection
>> is necessary.
> 
> Sagi, does this look good to you now?
> 

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[Sagi Grimberg]

>>>> Thanks Maurizio,
>>>>
>>>> Why do we need to the pf_cache.va protection? nvme_tcp_free_queue
>>>> is already protected against re-entry with the NVME_TCP_Q_ALLOCATED
>>>> bit.
>>>>
>>>
>>> It's not the re-entry what worried me, I thought that nvme_tcp_free_queue()
>>> might be called before page_frag_alloc() had the chance to initialize the pf_cache, triggering
>>> a NULL pointer dereference. I am doing some tests right now and it seems not to be
>>> possible so maybe we can drop the "if".
>>
>> Oh wait, if nvme_tcp_setup_ctrl() fails it could call nvme_tcp_destroy_io_queues() and
>> iI guess that in that case the pf_cache is not initialized, so the if(pf_cache.va) protection
>> is necessary.
> 
> Sagi, does this look good to you now?

Yes,

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>

Re: [PATCH] nvme-tcp: fix memory leak when freeing a queue

[Christoph Hellwig]

Thanks,

applied to nvme-5.16.