[PATCH] trace: convert init_trace_bufs() to constructor

[PATCH] trace: convert init_trace_bufs() to constructor

[Jan Beulich]

There's no need for each arch to invoke it directly, and there's no need
for having a stub either. With the present placement of the calls to
init_constructors() it can easily be a constructor itself.

Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
Same could then apparently be done for heap_init_late(). Thoughts?

--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -491,8 +491,6 @@ void asmlinkage __init start_xen(unsigne
 
     heap_init_late();
 
-    init_trace_bufs();
-
     init_constructors();
 
     console_endboot();
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -2143,8 +2143,6 @@ void asmlinkage __init noreturn __start_
 
     heap_init_late();
 
-    init_trace_bufs();
-
     init_constructors();
 
     console_endboot();
--- a/xen/common/trace.c
+++ b/xen/common/trace.c
@@ -336,7 +336,7 @@ int trace_will_trace_event(u32 event)
  * trace buffers.  The trace buffers are then available for debugging use, via
  * the %TRACE_xD macros exported in <xen/trace.h>.
  */
-void __init init_trace_bufs(void)
+static void __init __constructor init_trace_bufs(void)
 {
     cpumask_setall(&tb_cpu_mask);
     register_cpu_notifier(&cpu_nfb);
--- a/xen/include/xen/trace.h
+++ b/xen/include/xen/trace.h
@@ -29,9 +29,6 @@
 
 extern bool tb_init_done;
 
-/* Used to initialise trace buffer functionality */
-void init_trace_bufs(void);
-
 /* used to retrieve the physical address of the trace buffers */
 int tb_control(struct xen_sysctl_tbuf_op *tbc);
 
@@ -49,7 +46,6 @@ void __trace_hypercall(uint32_t event, u
 
 #define tb_init_done false
 
-static inline void init_trace_bufs(void) {}
 static inline int tb_control(struct xen_sysctl_tbuf_op *tbc)
 {
     return -ENOSYS;

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Andrew Cooper]

On 13/03/2025 1:38 pm, Jan Beulich wrote:
> There's no need for each arch to invoke it directly, and there's no need
> for having a stub either. With the present placement of the calls to
> init_constructors() it can easily be a constructor itself.
>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

This has a side effect of wiring it up on RISC-V and PPC, as they
process constructors.  It looks safe enough, but have you double checked?


However, the position and logic during init is nonsense, I think.

It registers a cpu notifier which only does spin_lock_init() on a
per-cpu variable, which I think only works today because 0 is the init
value.

alloc_trace_bufs() on the other hand has a for_each_online_cpu() loop
because it's too late and ought to be a presmp_initcall().

Also the allocations could be NUMA-local for all but the biggest of
servers (given the 16T upper limit because there are raw uint32_t's
involved in the protocol).

I'm tempted to ack this on the basis that it is an improvement, but a /*
TODO this is all mad, please fix */ wouldn't go amiss either.

~Andrew

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Jan Beulich]

On 13.03.2025 14:58, Andrew Cooper wrote:
> On 13/03/2025 1:38 pm, Jan Beulich wrote:
>> There's no need for each arch to invoke it directly, and there's no need
>> for having a stub either. With the present placement of the calls to
>> init_constructors() it can easily be a constructor itself.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> This has a side effect of wiring it up on RISC-V and PPC, as they
> process constructors.  It looks safe enough, but have you double checked?

I've been looking at this differently: For both it can't be right for the
function to _not_ be called. And no matter that ...

> However, the position and logic during init is nonsense, I think.
> 
> It registers a cpu notifier which only does spin_lock_init() on a
> per-cpu variable, which I think only works today because 0 is the init
> value.
> 
> alloc_trace_bufs() on the other hand has a for_each_online_cpu() loop
> because it's too late and ought to be a presmp_initcall().
> 
> Also the allocations could be NUMA-local for all but the biggest of
> servers (given the 16T upper limit because there are raw uint32_t's
> involved in the protocol).

... there's certainly further room for improvement, init_trace_bufs()
is all just "normal" code, which was already built before.

If there are missing pieces to make trace buffers fully working there,
that's no different from before the patch.

As to alloc_trace_bufs() - that has a 2nd caller, so converting to
presmp_initcall() may not buy us all that much.

> I'm tempted to ack this on the basis that it is an improvement, but a /*
> TODO this is all mad, please fix */ wouldn't go amiss either.

I understand you like adding such comments; I, however, at least
sometimes (e.g.) don't. Especially without at least outlining what
would need doing. Just saying "this is all mad" doesn't really help
very much.

Jan

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Andrew Cooper]

On 13/03/2025 2:19 pm, Jan Beulich wrote:
> On 13.03.2025 14:58, Andrew Cooper wrote:
>> On 13/03/2025 1:38 pm, Jan Beulich wrote:
>>> There's no need for each arch to invoke it directly, and there's no need
>>> for having a stub either. With the present placement of the calls to
>>> init_constructors() it can easily be a constructor itself.
>>>
>>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>> This has a side effect of wiring it up on RISC-V and PPC, as they
>> process constructors.  It looks safe enough, but have you double checked?
> I've been looking at this differently: For both it can't be right for the
> function to _not_ be called.

Eventually, sure.  But they're both in the early bringup stage, still
getting the basics working.

So really, the question is "does this (not) cause CI to explode".

In c/s 8c3ab4ffa953 I noted it was easy to make CONFIG_TRACEBUFFER build
for PPC, but I didn't try running init_trace_bufs().

Anyway, I've kicked off
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1715210166
to check.

>
>> However, the position and logic during init is nonsense, I think.
>>
>> It registers a cpu notifier which only does spin_lock_init() on a
>> per-cpu variable, which I think only works today because 0 is the init
>> value.
>>
>> alloc_trace_bufs() on the other hand has a for_each_online_cpu() loop
>> because it's too late and ought to be a presmp_initcall().
>>
>> Also the allocations could be NUMA-local for all but the biggest of
>> servers (given the 16T upper limit because there are raw uint32_t's
>> involved in the protocol).
> ... there's certainly further room for improvement, init_trace_bufs()
> is all just "normal" code, which was already built before.
>
> If there are missing pieces to make trace buffers fully working there,
> that's no different from before the patch.
>
> As to alloc_trace_bufs() - that has a 2nd caller, so converting to
> presmp_initcall() may not buy us all that much.

Another bug I've realised is that this fails if we hot-online new CPUs
later, because tb_init will be set but nothing allocated on the new CPUs.

>
>> I'm tempted to ack this on the basis that it is an improvement, but a /*
>> TODO this is all mad, please fix */ wouldn't go amiss either.
> I understand you like adding such comments; I, however, at least
> sometimes (e.g.) don't. Especially without at least outlining what
> would need doing. Just saying "this is all mad" doesn't really help
> very much.

I was being somewhat flippant.  But a /* TODO, try and make this a
presmp_initcall() to improve alloc_trace_bufs() */ would be fine.

~Andrew

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Jan Beulich]

On 13.03.2025 17:28, Andrew Cooper wrote:
> On 13/03/2025 2:19 pm, Jan Beulich wrote:
>> On 13.03.2025 14:58, Andrew Cooper wrote:
>>> On 13/03/2025 1:38 pm, Jan Beulich wrote:
>>> I'm tempted to ack this on the basis that it is an improvement, but a /*
>>> TODO this is all mad, please fix */ wouldn't go amiss either.
>> I understand you like adding such comments; I, however, at least
>> sometimes (e.g.) don't. Especially without at least outlining what
>> would need doing. Just saying "this is all mad" doesn't really help
>> very much.
> 
> I was being somewhat flippant.  But a /* TODO, try and make this a
> presmp_initcall() to improve alloc_trace_bufs() */ would be fine.

Okay, added (to the existing comment).

Jan

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Andrew Cooper]

On 13/03/2025 4:37 pm, Jan Beulich wrote:
> On 13.03.2025 17:28, Andrew Cooper wrote:
>> On 13/03/2025 2:19 pm, Jan Beulich wrote:
>>> On 13.03.2025 14:58, Andrew Cooper wrote:
>>>> On 13/03/2025 1:38 pm, Jan Beulich wrote:
>>>> I'm tempted to ack this on the basis that it is an improvement, but a /*
>>>> TODO this is all mad, please fix */ wouldn't go amiss either.
>>> I understand you like adding such comments; I, however, at least
>>> sometimes (e.g.) don't. Especially without at least outlining what
>>> would need doing. Just saying "this is all mad" doesn't really help
>>> very much.
>> I was being somewhat flippant.  But a /* TODO, try and make this a
>> presmp_initcall() to improve alloc_trace_bufs() */ would be fine.
> Okay, added (to the existing comment).

RISC-V and PPC were both green in the pipeline, so they seem happy.

~Andrew

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Jan Beulich]

On 13.03.2025 18:03, Andrew Cooper wrote:
> On 13/03/2025 4:37 pm, Jan Beulich wrote:
>> On 13.03.2025 17:28, Andrew Cooper wrote:
>>> On 13/03/2025 2:19 pm, Jan Beulich wrote:
>>>> On 13.03.2025 14:58, Andrew Cooper wrote:
>>>>> On 13/03/2025 1:38 pm, Jan Beulich wrote:
>>>>> I'm tempted to ack this on the basis that it is an improvement, but a /*
>>>>> TODO this is all mad, please fix */ wouldn't go amiss either.
>>>> I understand you like adding such comments; I, however, at least
>>>> sometimes (e.g.) don't. Especially without at least outlining what
>>>> would need doing. Just saying "this is all mad" doesn't really help
>>>> very much.
>>> I was being somewhat flippant.  But a /* TODO, try and make this a
>>> presmp_initcall() to improve alloc_trace_bufs() */ would be fine.
>> Okay, added (to the existing comment).
> 
> RISC-V and PPC were both green in the pipeline, so they seem happy.

As alluded to, not surprising at all, as the tests surely don't supply
a "tbuf_size=" command line option. Without which init_trace_bufs() does
close to nothing. Still - thanks for double checking. May I imply an ack
from this (formally I'll need a separate Arm one then still anyway)?

Jan

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Andrew Cooper]

On 14/03/2025 6:49 am, Jan Beulich wrote:
> On 13.03.2025 18:03, Andrew Cooper wrote:
>> On 13/03/2025 4:37 pm, Jan Beulich wrote:
>>> On 13.03.2025 17:28, Andrew Cooper wrote:
>>>> On 13/03/2025 2:19 pm, Jan Beulich wrote:
>>>>> On 13.03.2025 14:58, Andrew Cooper wrote:
>>>>>> On 13/03/2025 1:38 pm, Jan Beulich wrote:
>>>>>> I'm tempted to ack this on the basis that it is an improvement, but a /*
>>>>>> TODO this is all mad, please fix */ wouldn't go amiss either.
>>>>> I understand you like adding such comments; I, however, at least
>>>>> sometimes (e.g.) don't. Especially without at least outlining what
>>>>> would need doing. Just saying "this is all mad" doesn't really help
>>>>> very much.
>>>> I was being somewhat flippant.  But a /* TODO, try and make this a
>>>> presmp_initcall() to improve alloc_trace_bufs() */ would be fine.
>>> Okay, added (to the existing comment).
>> RISC-V and PPC were both green in the pipeline, so they seem happy.
> As alluded to, not surprising at all, as the tests surely don't supply
> a "tbuf_size=" command line option. Without which init_trace_bufs() does
> close to nothing. Still - thanks for double checking. May I imply an ack
> from this (formally I'll need a separate Arm one then still anyway)?

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Julien Grall]

Hi Jan,

On 13/03/2025 13:38, Jan Beulich wrote:
> There's no need for each arch to invoke it directly, and there's no need
> for having a stub either. With the present placement of the calls to
> init_constructors() it can easily be a constructor itself.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Acked-by: Julien Grall <jgrall@amazon.com>

> ---
> Same could then apparently be done for heap_init_late(). Thoughts?

Looking at the code, I couldn't figure out whether any of the 
constructors may rely on some changes done by heap_init_late().

The only issue I can think of is scrubbing. In the case it is 
synchronous, would the memory allocated before hand be scrubbed?

Cheers,

-- 
Julien Grall

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Jan Beulich]

On 27.03.2025 15:49, Julien Grall wrote:
> On 13/03/2025 13:38, Jan Beulich wrote:
>> There's no need for each arch to invoke it directly, and there's no need
>> for having a stub either. With the present placement of the calls to
>> init_constructors() it can easily be a constructor itself.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> Acked-by: Julien Grall <jgrall@amazon.com>

Thanks.

>> ---
>> Same could then apparently be done for heap_init_late(). Thoughts?
> 
> Looking at the code, I couldn't figure out whether any of the 
> constructors may rely on some changes done by heap_init_late().
> 
> The only issue I can think of is scrubbing. In the case it is 
> synchronous, would the memory allocated before hand be scrubbed?

Memory that is allocated can't possibly be scrubbed; only memory that's
still un-allocated can be. With that I fear I don't properly understand
the question you raise.

Jan

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Julien Grall]

Hi Jan,

On 27/03/2025 15:08, Jan Beulich wrote:
> On 27.03.2025 15:49, Julien Grall wrote:
>> On 13/03/2025 13:38, Jan Beulich wrote:
>>> There's no need for each arch to invoke it directly, and there's no need
>>> for having a stub either. With the present placement of the calls to
>>> init_constructors() it can easily be a constructor itself.
>>>
>>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>
>> Acked-by: Julien Grall <jgrall@amazon.com>
> 
> Thanks.
> 
>>> ---
>>> Same could then apparently be done for heap_init_late(). Thoughts?
>>
>> Looking at the code, I couldn't figure out whether any of the
>> constructors may rely on some changes done by heap_init_late().
>>
>> The only issue I can think of is scrubbing. In the case it is
>> synchronous, would the memory allocated before hand be scrubbed?
> 
> Memory that is allocated can't possibly be scrubbed; only memory that's
> still un-allocated can be. With that I fear I don't properly understand
> the question you raise.

I meant that if memory is allocated by calls from init_constructors(). 
Before this patch, the memory would be scrubbed. After this patch, 
anything constructors called before heap_init_late() would end up to not 
be scrubbed if it is synchronous.

Cheers,

-- 
Julien Grall

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Jan Beulich]

On 27.03.2025 16:49, Julien Grall wrote:
> On 27/03/2025 15:08, Jan Beulich wrote:
>> On 27.03.2025 15:49, Julien Grall wrote:
>>> On 13/03/2025 13:38, Jan Beulich wrote:
>>>> ---
>>>> Same could then apparently be done for heap_init_late(). Thoughts?
>>>
>>> Looking at the code, I couldn't figure out whether any of the
>>> constructors may rely on some changes done by heap_init_late().
>>>
>>> The only issue I can think of is scrubbing. In the case it is
>>> synchronous, would the memory allocated before hand be scrubbed?
>>
>> Memory that is allocated can't possibly be scrubbed; only memory that's
>> still un-allocated can be. With that I fear I don't properly understand
>> the question you raise.
> 
> I meant that if memory is allocated by calls from init_constructors(). 
> Before this patch, the memory would be scrubbed. After this patch, 
> anything constructors called before heap_init_late() would end up to not 
> be scrubbed if it is synchronous.

Oh, I see. Since scrubbing may be asynchronous, any site relying on scrubbing
having happened would be flawed anyway. Apart from that, unless callers pass
MEMF_no_scrub to alloc_heap_pages(), un-scrubbed pages would be scrubbed
anyway (see near the end of the function).

Jan

Re: [PATCH] trace: convert init_trace_bufs() to constructor

[Julien Grall]

Hi Jan,

On 27/03/2025 16:10, Jan Beulich wrote:
> On 27.03.2025 16:49, Julien Grall wrote:
>> On 27/03/2025 15:08, Jan Beulich wrote:
>>> On 27.03.2025 15:49, Julien Grall wrote:
>>>> On 13/03/2025 13:38, Jan Beulich wrote:
>>>>> ---
>>>>> Same could then apparently be done for heap_init_late(). Thoughts?
>>>>
>>>> Looking at the code, I couldn't figure out whether any of the
>>>> constructors may rely on some changes done by heap_init_late().
>>>>
>>>> The only issue I can think of is scrubbing. In the case it is
>>>> synchronous, would the memory allocated before hand be scrubbed?
>>>
>>> Memory that is allocated can't possibly be scrubbed; only memory that's
>>> still un-allocated can be. With that I fear I don't properly understand
>>> the question you raise.
>>
>> I meant that if memory is allocated by calls from init_constructors().
>> Before this patch, the memory would be scrubbed. After this patch,
>> anything constructors called before heap_init_late() would end up to not
>> be scrubbed if it is synchronous.
> 
> Oh, I see. Since scrubbing may be asynchronous, any site relying on scrubbing
> having happened would be flawed anyway. 

I have to disagree. If the scrubbing is asynchronous then...

> Apart from that, unless callers pass
> MEMF_no_scrub to alloc_heap_pages(), un-scrubbed pages would be scrubbed> anyway (see near the end of the function).

... the page will be scrubbed at the time it is allocated if it was not 
done before. But for synchronous scrubbing, at boot, this will not be 
the case (unless CONFIG_SCRUB_DEBUG is set and with the associated 
command line option). It will only happen during heap_init_late(). This 
is because init_heap_pages() will not request scrubbing unless 
asynchronous scrubbing is enabled (or CONFIG_SCRUB_DEBUG is set and with 
the associated command line).

So I still think there is a potential difference of behavior if we move 
heap_init_late() later. Someone will need to investigate whether there 
is effectively an issue.

Cheers,

-- 
Julien Grall