[PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Lee Jones]

From: Jakub Kicinski <kuba@kernel.org>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
may exit as soon as the async crypto handler calls complete().
Reorder scheduling the work before calling complete().
This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Reported-by: valis <sec@valis.email>
Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Simon Horman <horms@kernel.org>
Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
Signed-off-by: Lee Jones <lee@kernel.org>
---
 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 2bd27b77769cb..d53587ff9ddea 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -449,7 +449,6 @@ static void tls_encrypt_done(crypto_completion_data_t *data, int err)
 	struct scatterlist *sge;
 	struct sk_msg *msg_en;
 	struct tls_rec *rec;
-	bool ready = false;
 	struct sock *sk;
 
 	rec = container_of(aead_req, struct tls_rec, aead_req);
@@ -486,19 +485,16 @@ static void tls_encrypt_done(crypto_completion_data_t *data, int err)
 		/* If received record is at head of tx_list, schedule tx */
 		first_rec = list_first_entry(&ctx->tx_list,
 					     struct tls_rec, list);
-		if (rec == first_rec)
-			ready = true;
+		if (rec == first_rec) {
+			/* Schedule the transmission */
+			if (!test_and_set_bit(BIT_TX_SCHEDULED,
+					      &ctx->tx_bitmask))
+				schedule_delayed_work(&ctx->tx_work.work, 1);
+		}
 	}
 
 	if (atomic_dec_and_test(&ctx->encrypt_pending))
 		complete(&ctx->async_wait.completion);
-
-	if (!ready)
-		return;
-
-	/* Schedule the transmission */
-	if (!test_and_set_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask))
-		schedule_delayed_work(&ctx->tx_work.work, 1);
 }
 
 static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
-- 
2.44.0.278.ge034bb2e1d-goog

Re: [PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Jakub Kicinski]

On Thu,  7 Mar 2024 15:59:29 +0000 Lee Jones wrote:
> From: Jakub Kicinski <kuba@kernel.org>
> 
> [ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]
> 
> Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
> may exit as soon as the async crypto handler calls complete().
> Reorder scheduling the work before calling complete().
> This seems more logical in the first place, as it's
> the inverse order of what the submitting thread will do.
> 
> Reported-by: valis <sec@valis.email>
> Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> Reviewed-by: Simon Horman <horms@kernel.org>
> Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> Signed-off-by: Sasha Levin <sashal@kernel.org>
> (cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
> [Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
> Signed-off-by: Lee Jones <lee@kernel.org>

LG, thank you!

The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see
commit aec7961916f3f9e88766 in the other LTS branches. Without that
(it's still correct but) it doesn't fix the problem, because we still
touch the context after releasing the reference (unlocking the spin
lock).

Re: [PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Lee Jones]

On Thu, 07 Mar 2024, Jakub Kicinski wrote:

> On Thu,  7 Mar 2024 15:59:29 +0000 Lee Jones wrote:
> > From: Jakub Kicinski <kuba@kernel.org>
> > 
> > [ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]
> > 
> > Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
> > may exit as soon as the async crypto handler calls complete().
> > Reorder scheduling the work before calling complete().
> > This seems more logical in the first place, as it's
> > the inverse order of what the submitting thread will do.
> > 
> > Reported-by: valis <sec@valis.email>
> > Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
> > Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> > Reviewed-by: Simon Horman <horms@kernel.org>
> > Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
> > Signed-off-by: David S. Miller <davem@davemloft.net>
> > Signed-off-by: Sasha Levin <sashal@kernel.org>
> > (cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
> > [Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
> > Signed-off-by: Lee Jones <lee@kernel.org>
> 
> LG, thank you!
> 
> The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see
> commit aec7961916f3f9e88766 in the other LTS branches. Without that
> (it's still correct but) it doesn't fix the problem, because we still
> touch the context after releasing the reference (unlocking the spin
> lock).

No problem.

Should I accompany aec7961916f3 with this fix into the aforementioned
branches then?  Would that then be effective?

-- 
Lee Jones [李琼斯]

Re: [PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Jakub Kicinski]

On Thu, 7 Mar 2024 17:09:59 +0000 Lee Jones wrote:
> > The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see
> > commit aec7961916f3f9e88766 in the other LTS branches. Without that
> > (it's still correct but) it doesn't fix the problem, because we still
> > touch the context after releasing the reference (unlocking the spin
> > lock).  
> 
> No problem.
> 
> Should I accompany aec7961916f3 with this fix into the aforementioned
> branches then?  Would that then be effective?

Yes, (and c57ca512f3b68d, that's all the pre-req, I think.)
Tho, it may be a more tedious backport.
FWIW tools/testing/selftests/net/tls.c should be relatively solid.

Re: [PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Greg KH]

On Thu, Mar 07, 2024 at 03:59:29PM +0000, Lee Jones wrote:
> From: Jakub Kicinski <kuba@kernel.org>
> 
> [ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]
> 
> Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
> may exit as soon as the async crypto handler calls complete().
> Reorder scheduling the work before calling complete().
> This seems more logical in the first place, as it's
> the inverse order of what the submitting thread will do.
> 
> Reported-by: valis <sec@valis.email>
> Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> Reviewed-by: Simon Horman <horms@kernel.org>
> Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> Signed-off-by: Sasha Levin <sashal@kernel.org>
> (cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
> [Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
> Signed-off-by: Lee Jones <lee@kernel.org>
> ---
>  net/tls/tls_sw.c | 16 ++++++----------
>  1 file changed, 6 insertions(+), 10 deletions(-)
> 

Now qeueued up, but only this version, the older ones I've dropped from
my review queue based on the review from Jakub.  If they are still
needed, can you provide backported versions?

thanks,

greg k-h

Re: [PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Lee Jones]

On Wed, 27 Mar 2024, Greg KH wrote:

> On Thu, Mar 07, 2024 at 03:59:29PM +0000, Lee Jones wrote:
> > From: Jakub Kicinski <kuba@kernel.org>
> > 
> > [ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]
> > 
> > Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
> > may exit as soon as the async crypto handler calls complete().
> > Reorder scheduling the work before calling complete().
> > This seems more logical in the first place, as it's
> > the inverse order of what the submitting thread will do.
> > 
> > Reported-by: valis <sec@valis.email>
> > Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
> > Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> > Reviewed-by: Simon Horman <horms@kernel.org>
> > Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
> > Signed-off-by: David S. Miller <davem@davemloft.net>
> > Signed-off-by: Sasha Levin <sashal@kernel.org>
> > (cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
> > [Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
> > Signed-off-by: Lee Jones <lee@kernel.org>
> > ---
> >  net/tls/tls_sw.c | 16 ++++++----------
> >  1 file changed, 6 insertions(+), 10 deletions(-)
> > 
> 
> Now qeueued up, but only this version, the older ones I've dropped from
> my review queue based on the review from Jakub.  If they are still
> needed, can you provide backported versions?

Thanks.

Full disclosure, I have no plans to backport the remainder given Jakub's
comments.

-- 
Lee Jones [李琼斯]

Re: [PATCH 6.1 1/1] tls: fix race between tx work scheduling and socket close

[Harshit Mogalapalli]

Hi Jakub,

On 07/03/24 22:38, Jakub Kicinski wrote:
> On Thu,  7 Mar 2024 15:59:29 +0000 Lee Jones wrote:
>> From: Jakub Kicinski <kuba@kernel.org>
>>
>> [ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]
>>
>> Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
>> may exit as soon as the async crypto handler calls complete().
>> Reorder scheduling the work before calling complete().
>> This seems more logical in the first place, as it's
>> the inverse order of what the submitting thread will do.
>>
>> Reported-by: valis <sec@valis.email>
>> Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
>> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
>> Reviewed-by: Simon Horman <horms@kernel.org>
>> Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
>> Signed-off-by: David S. Miller <davem@davemloft.net>
>> Signed-off-by: Sasha Levin <sashal@kernel.org>
>> (cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
>> [Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
>> Signed-off-by: Lee Jones <lee@kernel.org>
> 
> LG, thank you!
> 
> The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see
> commit aec7961916f3f9e88766 in the other LTS branches. Without that
> (it's still correct but) it doesn't fix the problem, because we still
> touch the context after releasing the reference (unlocking the spin
> lock).
> 

commit: aec7961916f3 ("tls: fix race between async notify and socket 
close") is backported to 5.15.y as commit f17d21ea7391 ("tls: fix race 
between async notify and socket close") in v5.15.160

So I think we should now backport this fix e01e3934a1b2 ("tls: fix race 
between tx work scheduling and socket close") to 5.15.y ( this also 
fixes CVE-2024-26585)

Will send the backport to stable kernel list.

Thanks,
Harshit

[PATCH 5.15.y] tls: fix race between tx work scheduling and socket close

[Harshit Mogalapalli]

From: Jakub Kicinski <kuba@kernel.org>

commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb upstream.

Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
may exit as soon as the async crypto handler calls complete().
Reorder scheduling the work before calling complete().
This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Reported-by: valis <sec@valis.email>
Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Simon Horman <horms@kernel.org>
Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
Signed-off-by: Lee Jones <lee@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[Harshit: bp to 5.15.y, minor conflict resolutin due to missing commit:
8ae187386420 ("tls: Only use data field in crypto completion function")
in 5.15.y]
Signed-off-by: Harshit Mogalapalli <harshit.m.mogalapalli@oracle.com>
---
This is a fix for CVE-2024-26585, minor conflict resolution involved
Ran the tls self tests: 

ok 183 tls.13_chacha.shutdown_reuse
# PASSED: 183 / 183 tests passed.
# Totals: pass:183 fail:0 xfail:0 xpass:0 skip:0 error:0

 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 90f6cbe5cd5d..c17c3a14b9c1 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -468,7 +468,6 @@ static void tls_encrypt_done(struct crypto_async_request *req, int err)
 	struct scatterlist *sge;
 	struct sk_msg *msg_en;
 	struct tls_rec *rec;
-	bool ready = false;
 
 	if (err == -EINPROGRESS) /* see the comment in tls_decrypt_done() */
 		return;
@@ -502,19 +501,16 @@ static void tls_encrypt_done(struct crypto_async_request *req, int err)
 		/* If received record is at head of tx_list, schedule tx */
 		first_rec = list_first_entry(&ctx->tx_list,
 					     struct tls_rec, list);
-		if (rec == first_rec)
-			ready = true;
+		if (rec == first_rec) {
+			/* Schedule the transmission */
+			if (!test_and_set_bit(BIT_TX_SCHEDULED,
+					      &ctx->tx_bitmask))
+				schedule_delayed_work(&ctx->tx_work.work, 1);
+		}
 	}
 
 	if (atomic_dec_and_test(&ctx->encrypt_pending))
 		complete(&ctx->async_wait.completion);
-
-	if (!ready)
-		return;
-
-	/* Schedule the transmission */
-	if (!test_and_set_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask))
-		schedule_delayed_work(&ctx->tx_work.work, 1);
 }
 
 static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
-- 
2.45.2

Re: [PATCH 5.15.y] tls: fix race between tx work scheduling and socket close

[Greg Kroah-Hartman]

On Fri, Aug 02, 2024 at 04:39:31AM -0700, Harshit Mogalapalli wrote:
> From: Jakub Kicinski <kuba@kernel.org>
> 
> commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb upstream.
> 
> Similarly to previous commit, the submitting thread (recvmsg/sendmsg)
> may exit as soon as the async crypto handler calls complete().
> Reorder scheduling the work before calling complete().
> This seems more logical in the first place, as it's
> the inverse order of what the submitting thread will do.
> 
> Reported-by: valis <sec@valis.email>
> Fixes: a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> Reviewed-by: Simon Horman <horms@kernel.org>
> Reviewed-by: Sabrina Dubroca <sd@queasysnail.net>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> [Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
> Signed-off-by: Lee Jones <lee@kernel.org>
> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> [Harshit: bp to 5.15.y, minor conflict resolutin due to missing commit:
> 8ae187386420 ("tls: Only use data field in crypto completion function")
> in 5.15.y]
> Signed-off-by: Harshit Mogalapalli <harshit.m.mogalapalli@oracle.com>
> ---
> This is a fix for CVE-2024-26585, minor conflict resolution involved
> Ran the tls self tests: 
> 
> ok 183 tls.13_chacha.shutdown_reuse
> # PASSED: 183 / 183 tests passed.
> # Totals: pass:183 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
>  net/tls/tls_sw.c | 16 ++++++----------
>  1 file changed, 6 insertions(+), 10 deletions(-)

Now queued up, thanks.

greg k-h