[PATCH 0/2] target/i386: fix hang when using slow path for ptw_setl

[PATCH 0/2] target/i386: fix hang when using slow path for ptw_setl

[Pierrick Bouvier]

Most of the details are available in first patch. Second one is there to ensure
we'll have a useful error message if start_exclusive is called from cpu_exec
again.

I'm a bit puzzled that we never triggered this hang before. Is there something
wrong with the potential slow path for ptw_setl, or is it simply very uncommon?

Pierrick Bouvier (2):
  target/i386: fix hang when using slow path for ptw_setl
  cpu: ensure we don't call start_exclusive from cpu_exec

 cpu-common.c                         | 3 +++
 target/i386/tcg/sysemu/excp_helper.c | 4 ++++
 2 files changed, 7 insertions(+)

-- 
2.39.5

[PATCH 1/2] target/i386: fix hang when using slow path for ptw_setl

[Pierrick Bouvier]

When instrumenting memory accesses for plugin, we force memory accesses
to use the slow path for mmu. [1]
This create a situation where we end up calling ptw_setl_slow.

Since this function gets called during a cpu_exec, start_exclusive then
hangs. This exclusive section was introduced initially for security
reasons [2].

I suspect this code path was never triggered, because ptw_setl_slow
would always be called transitively from cpu_exec, resulting in a hang.

[1] https://gitlab.com/qemu-project/qemu/-/commit/6d03226b42247b68ab2f0b3663e0f624335a4055
[2] https://gitlab.com/qemu-project/qemu/-/issues/279

Fixes: https://gitlab.com/qemu-project/qemu/-/issues/2566
Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 target/i386/tcg/sysemu/excp_helper.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
index 8fb05b1f531..f30102b5362 100644
--- a/target/i386/tcg/sysemu/excp_helper.c
+++ b/target/i386/tcg/sysemu/excp_helper.c
@@ -108,6 +108,9 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
 {
     uint32_t cmp;
 
+    /* We are in cpu_exec, and start_exclusive can't be called directly.*/
+    g_assert(current_cpu && current_cpu->running);
+    cpu_exec_end(current_cpu);
     /* Does x86 really perform a rmw cycle on mmio for ptw? */
     start_exclusive();
     cmp = cpu_ldl_mmuidx_ra(in->env, in->gaddr, in->ptw_idx, 0);
@@ -115,6 +118,7 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
         cpu_stl_mmuidx_ra(in->env, in->gaddr, new, in->ptw_idx, 0);
     }
     end_exclusive();
+    cpu_exec_start(current_cpu);
     return cmp == old;
 }
 
-- 
2.39.5

Re: [PATCH 1/2] target/i386: fix hang when using slow path for ptw_setl

[Richard Henderson]

On 10/23/24 23:20, Pierrick Bouvier wrote:
> When instrumenting memory accesses for plugin, we force memory accesses
> to use the slow path for mmu. [1]
> This create a situation where we end up calling ptw_setl_slow.
> 
> Since this function gets called during a cpu_exec, start_exclusive then
> hangs. This exclusive section was introduced initially for security
> reasons [2].
> 
> I suspect this code path was never triggered, because ptw_setl_slow
> would always be called transitively from cpu_exec, resulting in a hang.
> 
> [1] https://gitlab.com/qemu-project/qemu/-/commit/6d03226b42247b68ab2f0b3663e0f624335a4055
> [2] https://gitlab.com/qemu-project/qemu/-/issues/279
> 
> Fixes: https://gitlab.com/qemu-project/qemu/-/issues/2566
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Oh, wow.  I believe this will be fixed by

https://lore.kernel.org/qemu-devel/20241023033432.1353830-19-richard.henderson@linaro.org/

which is in a pending PR.


r~


> ---
>   target/i386/tcg/sysemu/excp_helper.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
> index 8fb05b1f531..f30102b5362 100644
> --- a/target/i386/tcg/sysemu/excp_helper.c
> +++ b/target/i386/tcg/sysemu/excp_helper.c
> @@ -108,6 +108,9 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
>   {
>       uint32_t cmp;
>   
> +    /* We are in cpu_exec, and start_exclusive can't be called directly.*/
> +    g_assert(current_cpu && current_cpu->running);
> +    cpu_exec_end(current_cpu);
>       /* Does x86 really perform a rmw cycle on mmio for ptw? */
>       start_exclusive();
>       cmp = cpu_ldl_mmuidx_ra(in->env, in->gaddr, in->ptw_idx, 0);
> @@ -115,6 +118,7 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
>           cpu_stl_mmuidx_ra(in->env, in->gaddr, new, in->ptw_idx, 0);
>       }
>       end_exclusive();
> +    cpu_exec_start(current_cpu);
>       return cmp == old;
>   }
>   

Re: [PATCH 1/2] target/i386: fix hang when using slow path for ptw_setl

[Pierrick Bouvier]

On 10/24/24 09:25, Richard Henderson wrote:
> On 10/23/24 23:20, Pierrick Bouvier wrote:
>> When instrumenting memory accesses for plugin, we force memory accesses
>> to use the slow path for mmu. [1]
>> This create a situation where we end up calling ptw_setl_slow.
>>
>> Since this function gets called during a cpu_exec, start_exclusive then
>> hangs. This exclusive section was introduced initially for security
>> reasons [2].
>>
>> I suspect this code path was never triggered, because ptw_setl_slow
>> would always be called transitively from cpu_exec, resulting in a hang.
>>
>> [1] https://gitlab.com/qemu-project/qemu/-/commit/6d03226b42247b68ab2f0b3663e0f624335a4055
>> [2] https://gitlab.com/qemu-project/qemu/-/issues/279
>>
>> Fixes: https://gitlab.com/qemu-project/qemu/-/issues/2566
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> 
> Oh, wow.  I believe this will be fixed by
> 
> https://lore.kernel.org/qemu-devel/20241023033432.1353830-19-richard.henderson@linaro.org/
> 
> which is in a pending PR.
> 

It might the issue by not triggering the situation we observed.
However, we still have a hidden dead code path where start_exclusive is 
called from cpu_exec, not being related to the plugins.

> 
> r~
> 
> 
>> ---
>>    target/i386/tcg/sysemu/excp_helper.c | 4 ++++
>>    1 file changed, 4 insertions(+)
>>
>> diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
>> index 8fb05b1f531..f30102b5362 100644
>> --- a/target/i386/tcg/sysemu/excp_helper.c
>> +++ b/target/i386/tcg/sysemu/excp_helper.c
>> @@ -108,6 +108,9 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
>>    {
>>        uint32_t cmp;
>>    
>> +    /* We are in cpu_exec, and start_exclusive can't be called directly.*/
>> +    g_assert(current_cpu && current_cpu->running);
>> +    cpu_exec_end(current_cpu);
>>        /* Does x86 really perform a rmw cycle on mmio for ptw? */
>>        start_exclusive();
>>        cmp = cpu_ldl_mmuidx_ra(in->env, in->gaddr, in->ptw_idx, 0);
>> @@ -115,6 +118,7 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
>>            cpu_stl_mmuidx_ra(in->env, in->gaddr, new, in->ptw_idx, 0);
>>        }
>>        end_exclusive();
>> +    cpu_exec_start(current_cpu);
>>        return cmp == old;
>>    }
>>    
> 

Re: [PATCH 1/2] target/i386: fix hang when using slow path for ptw_setl

[Richard Henderson]

On 10/24/24 18:14, Pierrick Bouvier wrote:
> On 10/24/24 09:25, Richard Henderson wrote:
>> On 10/23/24 23:20, Pierrick Bouvier wrote:
>>> When instrumenting memory accesses for plugin, we force memory accesses
>>> to use the slow path for mmu. [1]
>>> This create a situation where we end up calling ptw_setl_slow.
>>>
>>> Since this function gets called during a cpu_exec, start_exclusive then
>>> hangs. This exclusive section was introduced initially for security
>>> reasons [2].
>>>
>>> I suspect this code path was never triggered, because ptw_setl_slow
>>> would always be called transitively from cpu_exec, resulting in a hang.
>>>
>>> [1] https://gitlab.com/qemu-project/qemu/-/commit/6d03226b42247b68ab2f0b3663e0f624335a4055
>>> [2] https://gitlab.com/qemu-project/qemu/-/issues/279
>>>
>>> Fixes: https://gitlab.com/qemu-project/qemu/-/issues/2566
>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>
>> Oh, wow.  I believe this will be fixed by
>>
>> https://lore.kernel.org/qemu-devel/20241023033432.1353830-19-richard.henderson@linaro.org/
>>
>> which is in a pending PR.
>>
> 
> It might the issue by not triggering the situation we observed.
> However, we still have a hidden dead code path where start_exclusive is called from 
> cpu_exec, not being related to the plugins.

You're right, this would affect mmio, were the os so careless as to place page tables in mmio.

>>> +    /* We are in cpu_exec, and start_exclusive can't be called directly.*/
>>> +    g_assert(current_cpu && current_cpu->running);
>>> +    cpu_exec_end(current_cpu);

Better to use env_cpu(in->env).


r~

Re: [PATCH 1/2] target/i386: fix hang when using slow path for ptw_setl

[Pierrick Bouvier]

On 10/24/24 09:25, Richard Henderson wrote:
> On 10/23/24 23:20, Pierrick Bouvier wrote:
>> When instrumenting memory accesses for plugin, we force memory accesses
>> to use the slow path for mmu. [1]
>> This create a situation where we end up calling ptw_setl_slow.
>>
>> Since this function gets called during a cpu_exec, start_exclusive then
>> hangs. This exclusive section was introduced initially for security
>> reasons [2].
>>
>> I suspect this code path was never triggered, because ptw_setl_slow
>> would always be called transitively from cpu_exec, resulting in a hang.
>>
>> [1] https://gitlab.com/qemu-project/qemu/-/commit/6d03226b42247b68ab2f0b3663e0f624335a4055
>> [2] https://gitlab.com/qemu-project/qemu/-/issues/279
>>
>> Fixes: https://gitlab.com/qemu-project/qemu/-/issues/2566
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> 
> Oh, wow.  I believe this will be fixed by
> 
> https://lore.kernel.org/qemu-devel/20241023033432.1353830-19-richard.henderson@linaro.org/
> 
> which is in a pending PR.
> 

I confirm this fix the issue and it's now merged upstream.

> 
> r~
> 
> 
>> ---
>>    target/i386/tcg/sysemu/excp_helper.c | 4 ++++
>>    1 file changed, 4 insertions(+)
>>
>> diff --git a/target/i386/tcg/sysemu/excp_helper.c b/target/i386/tcg/sysemu/excp_helper.c
>> index 8fb05b1f531..f30102b5362 100644
>> --- a/target/i386/tcg/sysemu/excp_helper.c
>> +++ b/target/i386/tcg/sysemu/excp_helper.c
>> @@ -108,6 +108,9 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
>>    {
>>        uint32_t cmp;
>>    
>> +    /* We are in cpu_exec, and start_exclusive can't be called directly.*/
>> +    g_assert(current_cpu && current_cpu->running);
>> +    cpu_exec_end(current_cpu);
>>        /* Does x86 really perform a rmw cycle on mmio for ptw? */
>>        start_exclusive();
>>        cmp = cpu_ldl_mmuidx_ra(in->env, in->gaddr, in->ptw_idx, 0);
>> @@ -115,6 +118,7 @@ static bool ptw_setl_slow(const PTETranslate *in, uint32_t old, uint32_t new)
>>            cpu_stl_mmuidx_ra(in->env, in->gaddr, new, in->ptw_idx, 0);
>>        }
>>        end_exclusive();
>> +    cpu_exec_start(current_cpu);
>>        return cmp == old;
>>    }
>>    
> 

[PATCH 2/2] cpu: ensure we don't call start_exclusive from cpu_exec

[Pierrick Bouvier]

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 cpu-common.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/cpu-common.c b/cpu-common.c
index 6b262233a3b..c979138fce9 100644
--- a/cpu-common.c
+++ b/cpu-common.c
@@ -194,6 +194,9 @@ void start_exclusive(void)
     CPUState *other_cpu;
     int running_cpus;
 
+    /* Ensure we are not running, or start_exclusive will be blocked. */
+    g_assert(!current_cpu || !current_cpu->running);
+
     if (current_cpu->exclusive_context_count) {
         current_cpu->exclusive_context_count++;
         return;
-- 
2.39.5

Re: [PATCH 2/2] cpu: ensure we don't call start_exclusive from cpu_exec

[Richard Henderson]

On 10/23/24 23:20, Pierrick Bouvier wrote:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>   cpu-common.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/cpu-common.c b/cpu-common.c
> index 6b262233a3b..c979138fce9 100644
> --- a/cpu-common.c
> +++ b/cpu-common.c
> @@ -194,6 +194,9 @@ void start_exclusive(void)
>       CPUState *other_cpu;
>       int running_cpus;
>   
> +    /* Ensure we are not running, or start_exclusive will be blocked. */
> +    g_assert(!current_cpu || !current_cpu->running);
> +
>       if (current_cpu->exclusive_context_count) {
>           current_cpu->exclusive_context_count++;
>           return;

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH 2/2] cpu: ensure we don't call start_exclusive from cpu_exec

[Richard Henderson]

On 10/23/24 23:20, Pierrick Bouvier wrote:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>   cpu-common.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/cpu-common.c b/cpu-common.c
> index 6b262233a3b..c979138fce9 100644
> --- a/cpu-common.c
> +++ b/cpu-common.c
> @@ -194,6 +194,9 @@ void start_exclusive(void)
>       CPUState *other_cpu;
>       int running_cpus;
>   
> +    /* Ensure we are not running, or start_exclusive will be blocked. */
> +    g_assert(!current_cpu || !current_cpu->running);
> +
>       if (current_cpu->exclusive_context_count) {
>           current_cpu->exclusive_context_count++;
>           return;

Actually, current_cpu had better be non-null from the very next line.


r~