[PATCH 0/2] Fix some kernel panics due to the host sending invalid PDUs

[PATCH 0/2] Fix some kernel panics due to the host sending invalid PDUs

[Maurizio Lombardi]

I used the reproducer that was published here:
https://lore.kernel.org/linux-nvme/89a927a6-2baf-434a-b1d5-50fb99beca73@grimberg.me/T/#m57f25abc944dc706125fed02cb99d86b56c5d36e

I didn't get the same call trace, however.

Maurizio Lombardi (2):
  nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU
    length
  nvmet-tcp: fix a crash in nvmet_req_complete()

 drivers/nvme/target/tcp.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

-- 
2.39.3

[PATCH 1/2] nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU length

[Maurizio Lombardi]

If the host sends an H2CData command with a DATAL field equal to 0,
the kernel crashes in nvmet_tcp_build_pdu_iovec().

Unable to handle kernel NULL pointer dereference at
virtual address 0000000000000000
lr : nvmet_tcp_io_work+0x6ac/0x718 [nvmet_tcp]
Call trace:
  process_one_work+0x174/0x3c8
  worker_thread+0x2d0/0x3e8
  kthread+0x104/0x110

Getting an H2CData Packet with DATAL=0 is unexpected, treat it
like a fatal transport error.
Also, the PDU length should never exceed the MAXH2CDATA parameter which
has been communicated to the host in nvmet_tcp_handle_icreq().

Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
---
 drivers/nvme/target/tcp.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 4cc27856aa8f..44eda6081e99 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -24,6 +24,7 @@
 #include "nvmet.h"
 
 #define NVMET_TCP_DEF_INLINE_DATA_SIZE	(4 * PAGE_SIZE)
+#define NVMET_TCP_MAXH2CDATA		0x400000 /* 16M arbitrary limit */
 
 static int param_store_val(const char *str, int *val, int min, int max)
 {
@@ -923,7 +924,7 @@ static int nvmet_tcp_handle_icreq(struct nvmet_tcp_queue *queue)
 	icresp->hdr.pdo = 0;
 	icresp->hdr.plen = cpu_to_le32(icresp->hdr.hlen);
 	icresp->pfv = cpu_to_le16(NVME_TCP_PFV_1_0);
-	icresp->maxdata = cpu_to_le32(0x400000); /* 16M arbitrary limit */
+	icresp->maxdata = cpu_to_le32(NVMET_TCP_MAXH2CDATA);
 	icresp->cpda = 0;
 	if (queue->hdr_digest)
 		icresp->digest |= NVME_TCP_HDR_DIGEST_ENABLE;
@@ -1002,6 +1003,13 @@ static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue)
 	}
 
 	cmd->pdu_len = le32_to_cpu(data->data_length);
+	if (unlikely(cmd->pdu_len == 0 ||
+		     cmd->pdu_len > NVMET_TCP_MAXH2CDATA)) {
+		pr_err("H2CData PDU len out of range\n");
+		/* FIXME: use proper transport errors */
+		nvmet_tcp_fatal_error(queue);
+		return -EPROTO;
+	}
 	cmd->pdu_recv = 0;
 	nvmet_tcp_build_pdu_iovec(cmd);
 	queue->cmd = cmd;
-- 
2.39.3

Re: [PATCH 1/2] nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU length

[Keith Busch]

On Thu, Dec 21, 2023 at 05:31:53PM +0100, Maurizio Lombardi wrote:
> @@ -1002,6 +1003,13 @@ static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue)
>  	}
>  
>  	cmd->pdu_len = le32_to_cpu(data->data_length);
> +	if (unlikely(cmd->pdu_len == 0 ||
> +		     cmd->pdu_len > NVMET_TCP_MAXH2CDATA)) {
> +		pr_err("H2CData PDU len out of range\n");

Please add the 'pdu_len' to the pr_err since that info sounds useful to
knowing which type of target error we're dealing with.

Otherwise looks good!

Reviewed-by: Keith Busch <kbusch@kernel.org>

> +		/* FIXME: use proper transport errors */
> +		nvmet_tcp_fatal_error(queue);
> +		return -EPROTO;
> +	}
>  	cmd->pdu_recv = 0;
>  	nvmet_tcp_build_pdu_iovec(cmd);
>  	queue->cmd = cmd;

Re: [PATCH 1/2] nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU length

[Maurizio Lombardi]

čt 21. 12. 2023 v 18:42 odesílatel Keith Busch <kbusch@kernel.org> napsal:
>
> On Thu, Dec 21, 2023 at 05:31:53PM +0100, Maurizio Lombardi wrote:
> Please add the 'pdu_len' to the pr_err since that info sounds useful to
> knowing which type of target error we're dealing with.
>
> Otherwise looks good!
>
> Reviewed-by: Keith Busch <kbusch@kernel.org>
>

Thanks,

I realized that there is still a possible problem, the data_length
sent by the host
could be inside the allowed range, but still higher than the real size
of the packet.
This could make the kernel crash.

Today I will send a V2 and add more fixes to this patchset.

Maurizio

[PATCH 2/2] nvmet-tcp: fix a crash in nvmet_req_complete()

[Maurizio Lombardi]

in nvmet_tcp_handle_h2c_data_pdu(), if the host sends a data_offset
different from rbytes_done, the driver ends up calling nvmet_req_complete()
passing a status error.
The problem is that at this point cmd->req is not yet initialized,
the kernel will crash after dereferencing a NULL pointer.

Fix the bug by replacing the call to nvmet_req_complete() with
nvmet_tcp_fatal_error().

Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
---
 drivers/nvme/target/tcp.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 44eda6081e99..0e719460199f 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -997,8 +997,7 @@ static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue)
 			data->ttag, le32_to_cpu(data->data_offset),
 			cmd->rbytes_done);
 		/* FIXME: use path and transport errors */
-		nvmet_req_complete(&cmd->req,
-			NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+		nvmet_tcp_fatal_error(queue);
 		return -EPROTO;
 	}
 
-- 
2.39.3

Re: [PATCH 2/2] nvmet-tcp: fix a crash in nvmet_req_complete()

[Keith Busch]

On Thu, Dec 21, 2023 at 05:31:54PM +0100, Maurizio Lombardi wrote:
> in nvmet_tcp_handle_h2c_data_pdu(), if the host sends a data_offset
> different from rbytes_done, the driver ends up calling nvmet_req_complete()
> passing a status error.
> The problem is that at this point cmd->req is not yet initialized,
> the kernel will crash after dereferencing a NULL pointer.
> 
> Fix the bug by replacing the call to nvmet_req_complete() with
> nvmet_tcp_fatal_error().

Looks good. The bug this fixes goes back to the beginning:

Fixes: 872d26a391da92 ("nvmet-tcp: add NVMe over TCP target driver")

Reviewed-by: Keith Busch <kbsuch@kernel.org>