[PATCH] x86/sgx: Avoid using iterator after loop in sgx_mmu_notifier_release()

[PATCH] x86/sgx: Avoid using iterator after loop in sgx_mmu_notifier_release()

[Jakob Koschel]

If &encl_mm->encl->mm_list does not contain the searched 'encl_mm',
'tmp' will not point to a valid sgx_encl_mm struct.

Since the code within the guarded block is just called when the element
is found, it can simply be moved into the list iterator.
Within the list iterator 'tmp' is guaranteed to point to a valid
element.

Signed-off-by: Jakob Koschel <jkl820.git@gmail.com>
---
Linus proposed to avoid any use of the list iterator variable after the
loop, in the attempt to move the list iterator variable declaration into
the marcro to avoid any potential misuse after the loop.
Using it in a pointer comparision after the loop is undefined behavior
and should be omitted if possible [1].

Link: https://lore.kernel.org/all/CAHk-=wgRr_D8CB-D9Kg-c=EHreAsk5SqXPwr9Y7k9sA6cWXJ6w@mail.gmail.com/ [1]
---
 arch/x86/kernel/cpu/sgx/encl.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
index 2a0e90fe2abc..db585b780141 100644
--- a/arch/x86/kernel/cpu/sgx/encl.c
+++ b/arch/x86/kernel/cpu/sgx/encl.c
@@ -764,15 +764,13 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
 	list_for_each_entry(tmp, &encl_mm->encl->mm_list, list) {
 		if (tmp == encl_mm) {
 			list_del_rcu(&encl_mm->list);
-			break;
+			spin_unlock(&encl_mm->encl->mm_lock);
+			synchronize_srcu(&encl_mm->encl->srcu);
+			mmu_notifier_put(mn);
+			return;
 		}
 	}
 	spin_unlock(&encl_mm->encl->mm_lock);
-
-	if (tmp == encl_mm) {
-		synchronize_srcu(&encl_mm->encl->srcu);
-		mmu_notifier_put(mn);
-	}
 }
 
 static void sgx_mmu_notifier_free(struct mmu_notifier *mn)

---
base-commit: d2d11f342b179f1894a901f143ec7c008caba43e
change-id: 20230206-sgx-use-after-iter-f584c1d64c87

Best regards,
-- 
Jakob Koschel <jkl820.git@gmail.com>

Re: [PATCH] x86/sgx: Avoid using iterator after loop in sgx_mmu_notifier_release()

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On 2/6/23 02:39, Jakob Koschel wrote:
> If &encl_mm->encl->mm_list does not contain the searched 'encl_mm',
> 'tmp' will not point to a valid sgx_encl_mm struct.
> 
> Since the code within the guarded block is just called when the element
> is found, it can simply be moved into the list iterator.
> Within the list iterator 'tmp' is guaranteed to point to a valid
> element.
> 
> Signed-off-by: Jakob Koschel <jkl820.git@gmail.com>
> ---
> Linus proposed to avoid any use of the list iterator variable after the
> loop, in the attempt to move the list iterator variable declaration into
> the marcro to avoid any potential misuse after the loop.
> Using it in a pointer comparision after the loop is undefined behavior
> and should be omitted if possible [1].

I think there's a big difference between "undefined behavior" and
"someone wants to flip a switch to *make* this undefined behavior".  My
understanding is that this patch avoids behavior which _is_ defined today.

Is there some effort to change this behavior across the tree that I missed?

In any case, this patch also kinda breaks the rule that you're supposed
to make the common path through the code at the lowest nesting level.
It makes the common case look like some kind of error handling.  Would
something like the attached patch work?

[-- Attachment #2: sgxmm.patch --]
[-- Type: text/x-patch, Size: 900 bytes --]

diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
index 68f8b18d2278..e1bd2a5790a7 100644
--- a/arch/x86/kernel/cpu/sgx/encl.c
+++ b/arch/x86/kernel/cpu/sgx/encl.c
@@ -755,6 +755,7 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
 {
 	struct sgx_encl_mm *encl_mm = container_of(mn, struct sgx_encl_mm, mmu_notifier);
 	struct sgx_encl_mm *tmp = NULL;
+	bool mm_found = false;
 
 	/*
 	 * The enclave itself can remove encl_mm.  Note, objects can't be moved
@@ -764,12 +765,13 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
 	list_for_each_entry(tmp, &encl_mm->encl->mm_list, list) {
 		if (tmp == encl_mm) {
 			list_del_rcu(&encl_mm->list);
+			mm_found = true;
 			break;
 		}
 	}
 	spin_unlock(&encl_mm->encl->mm_lock);
 
-	if (tmp == encl_mm) {
+	if (mm_found) {
 		synchronize_srcu(&encl_mm->encl->srcu);
 		mmu_notifier_put(mn);
 	}

Re: [PATCH] x86/sgx: Avoid using iterator after loop in sgx_mmu_notifier_release()

[Jakob Koschel]

> On 6. Feb 2023, at 18:10, Dave Hansen <dave.hansen@intel.com> wrote:
> 
> On 2/6/23 02:39, Jakob Koschel wrote:
>> If &encl_mm->encl->mm_list does not contain the searched 'encl_mm',
>> 'tmp' will not point to a valid sgx_encl_mm struct.
>> 
>> Since the code within the guarded block is just called when the element
>> is found, it can simply be moved into the list iterator.
>> Within the list iterator 'tmp' is guaranteed to point to a valid
>> element.
>> 
>> Signed-off-by: Jakob Koschel <jkl820.git@gmail.com>
>> ---
>> Linus proposed to avoid any use of the list iterator variable after the
>> loop, in the attempt to move the list iterator variable declaration into
>> the marcro to avoid any potential misuse after the loop.
>> Using it in a pointer comparision after the loop is undefined behavior
>> and should be omitted if possible [1].
> 
> I think there's a big difference between "undefined behavior" and
> "someone wants to flip a switch to *make* this undefined behavior".  My
> understanding is that this patch avoids behavior which _is_ defined today.
> 
> Is there some effort to change this behavior across the tree that I missed?

Some background information if interested: 

If 'encl_mm' was not found, 'tmp' always points somewhere into 'sgx_encl'.
In that case it is not possible to match 'encl_mm' (if 'encl_mm' always
points to a valid struct).

It's still used as a 'struct sgx_encl_mm' pointer even though the memory
it is pointing to is something completely different, which I would argue
is undefined behavior. 
(I would argue not all undefined behavior is automatically exploitable or
causing a bug)

[1] shows some effort on removing any use of the list iterator variable
after the loop, so ideally it can be declared within the macro itself.
I've showcased a similar case to this becoming an issue after reordering
certain struct members in [2].

Some more information here [3], [4].

> 
> In any case, this patch also kinda breaks the rule that you're supposed
> to make the common path through the code at the lowest nesting level.
> It makes the common case look like some kind of error handling.  Would
> something like the attached patch work?<sgxmm.patch>

of course! Actually the way I would have preferred.

I've posted several patches fixing those cases all across the tree and
always got mixed feedback. 
(I proposed your way to others and was asked to change it into the
way I posted it and vice versa).

I'm happy to change it to the way you proposed way in v2 ;) Thanks!

- Jakob

Link: https://lore.kernel.org/all/CAHk-=wgRr_D8CB-D9Kg-c=EHreAsk5SqXPwr9Y7k9sA6cWXJ6w@mail.gmail.com/ [1]
Link: https://lore.kernel.org/all/86C4CE7D-6D93-456B-AA82-F8ADEACA40B7@gmail.com/ [2]
Link: https://lwn.net/Articles/885941/ [3]
Link: https://lwn.net/Articles/887097/ [4]

Re: [PATCH] x86/sgx: Avoid using iterator after loop in sgx_mmu_notifier_release()

[Jarkko Sakkinen]

On Mon, Feb 06, 2023 at 09:10:53AM -0800, Dave Hansen wrote:
> On 2/6/23 02:39, Jakob Koschel wrote:
> > If &encl_mm->encl->mm_list does not contain the searched 'encl_mm',
> > 'tmp' will not point to a valid sgx_encl_mm struct.
> > 
> > Since the code within the guarded block is just called when the element
> > is found, it can simply be moved into the list iterator.
> > Within the list iterator 'tmp' is guaranteed to point to a valid
> > element.
> > 
> > Signed-off-by: Jakob Koschel <jkl820.git@gmail.com>
> > ---
> > Linus proposed to avoid any use of the list iterator variable after the
> > loop, in the attempt to move the list iterator variable declaration into
> > the marcro to avoid any potential misuse after the loop.
> > Using it in a pointer comparision after the loop is undefined behavior
> > and should be omitted if possible [1].
> 
> I think there's a big difference between "undefined behavior" and
> "someone wants to flip a switch to *make* this undefined behavior".  My
> understanding is that this patch avoids behavior which _is_ defined today.
> 
> Is there some effort to change this behavior across the tree that I missed?
> 
> In any case, this patch also kinda breaks the rule that you're supposed
> to make the common path through the code at the lowest nesting level.
> It makes the common case look like some kind of error handling.  Would
> something like the attached patch work?

> diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
> index 68f8b18d2278..e1bd2a5790a7 100644
> --- a/arch/x86/kernel/cpu/sgx/encl.c
> +++ b/arch/x86/kernel/cpu/sgx/encl.c
> @@ -755,6 +755,7 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
>  {
>  	struct sgx_encl_mm *encl_mm = container_of(mn, struct sgx_encl_mm, mmu_notifier);
>  	struct sgx_encl_mm *tmp = NULL;
> +	bool mm_found = false;

Maybe just "found" ? (nit)

>  
>  	/*
>  	 * The enclave itself can remove encl_mm.  Note, objects can't be moved
> @@ -764,12 +765,13 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
>  	list_for_each_entry(tmp, &encl_mm->encl->mm_list, list) {
>  		if (tmp == encl_mm) {
>  			list_del_rcu(&encl_mm->list);
> +			mm_found = true;
>  			break;
>  		}
>  	}
>  	spin_unlock(&encl_mm->encl->mm_lock);
>  
> -	if (tmp == encl_mm) {
> +	if (mm_found) {
>  		synchronize_srcu(&encl_mm->encl->srcu);
>  		mmu_notifier_put(mn);
>  	}

BR, Jarkko

Re: [PATCH] x86/sgx: Avoid using iterator after loop in sgx_mmu_notifier_release()

[Jarkko Sakkinen]

On Mon, Feb 06, 2023 at 11:39:02AM +0100, Jakob Koschel wrote:
> If &encl_mm->encl->mm_list does not contain the searched 'encl_mm',
> 'tmp' will not point to a valid sgx_encl_mm struct.

Please explain.

Perhaps, extend with something like ", and can be turned into
transient gadget."

> 
> Since the code within the guarded block is just called when the element
> is found, it can simply be moved into the list iterator.
> Within the list iterator 'tmp' is guaranteed to point to a valid
> element.
> 
> Signed-off-by: Jakob Koschel <jkl820.git@gmail.com>
> ---
> Linus proposed to avoid any use of the list iterator variable after the
> loop, in the attempt to move the list iterator variable declaration into
> the marcro to avoid any potential misuse after the loop.
> Using it in a pointer comparision after the loop is undefined behavior
> and should be omitted if possible [1].
> 
> Link: https://lore.kernel.org/all/CAHk-=wgRr_D8CB-D9Kg-c=EHreAsk5SqXPwr9Y7k9sA6cWXJ6w@mail.gmail.com/ [1]

I would move all this to the commit message. It is useful information.

> ---
>  arch/x86/kernel/cpu/sgx/encl.c | 10 ++++------
>  1 file changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
> index 2a0e90fe2abc..db585b780141 100644
> --- a/arch/x86/kernel/cpu/sgx/encl.c
> +++ b/arch/x86/kernel/cpu/sgx/encl.c
> @@ -764,15 +764,13 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
>  	list_for_each_entry(tmp, &encl_mm->encl->mm_list, list) {
>  		if (tmp == encl_mm) {
>  			list_del_rcu(&encl_mm->list);
> -			break;
> +			spin_unlock(&encl_mm->encl->mm_lock);
> +			synchronize_srcu(&encl_mm->encl->srcu);
> +			mmu_notifier_put(mn);
> +			return;
>  		}
>  	}
>  	spin_unlock(&encl_mm->encl->mm_lock);
> -
> -	if (tmp == encl_mm) {
> -		synchronize_srcu(&encl_mm->encl->srcu);
> -		mmu_notifier_put(mn);
> -	}
>  }
>  
>  static void sgx_mmu_notifier_free(struct mmu_notifier *mn)
> 
> ---
> base-commit: d2d11f342b179f1894a901f143ec7c008caba43e
> change-id: 20230206-sgx-use-after-iter-f584c1d64c87
> 
> Best regards,
> -- 
> Jakob Koschel <jkl820.git@gmail.com>
> 

BR, Jarkko