Re: [RFC 0/4] Transparent on-demand struct page initialization embedded in the buddy allocator

Ingo Molnar
2013-07-12T08:28:02Z

* Robin Holt <holt@sgi.com> wrote:

> [...]
>
> With this patch, we did boot a 16TiB machine. Without the patches, the
> v3.10 kernel with the same configuration took 407 seconds for
> free_all_bootmem. With the patches and operating on 2MiB pages instead
> of 1GiB, it took 26 seconds so performance was improved. I have no feel
> for how the 1GiB chunk size will perform.

That's pretty impressive.

It's still a 15x speedup instead of a 512x speedup, so I'd say there's
something else being the current bottleneck, besides page init
granularity.

Can you boot with just a few gigs of RAM and stuff the rest into hotplug
memory, and then hot-add that memory? That would allow easy profiling of
remaining overhead.

Side note:

Robert Richter and Boris Petkov are working on 'persistent events' support
for perf, which will eventually allow boot time profiling - I'm not sure
if the patches and the tooling support is ready enough yet for your
purposes.

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

- we'd get a single (cycles?) event running once the perf subsystem is up
and running, with a sampling frequency of 1 KHz, sending profiling
trace events to a sufficiently sized profiling buffer of 16 MB per
CPU.

- once the system reaches SYSTEM_RUNNING, profiling is stopped either
automatically - or the user stops it via a new tooling command.

- the profiling buffer is extracted into a regular perf.data via a
special 'perf record' call or some other, new perf tooling
solution/variant.

[ Alternatively the kernel could attempt to construct a 'virtual'
perf.data from the persistent buffer, available via /sys/debug or
elsewhere in /sys - just like the kernel constructs a 'virtual'
/proc/kcore, etc. That file could be copied or used directly. ]

- from that point on this workflow joins the regular profiling workflow:
perf report, perf script et al can be used to analyze the resulting
boot profile.

Thanks,

Ingo