* [boot-time] jent_mod_init on beagleplay (was RE: [boot-time] [RFC] analyze-initcall-debug.py - a tool to analyze the initcall debug output
From: Bird, Tim @ 2024-12-03 21:24 UTC (permalink / raw)
To: Francesco Valla, Linux Embedded; +Cc: smueller@chronox.de
In-Reply-To: <1964175.7Z3S40VBb9@fedora.fritz.box>
> -----Original Message-----
> From: Francesco Valla <francesco@valla.it>
> Dear fellow boot time optimizers,
>
> following the first boot time SIG meeting, which I lurked with much pleasure
> (but didn't participate to, as I was a bit in awe among such authorities), I'd
> like to introduce myself with code rather than a presentation or resume.
>
> Here is a python script which can analyze a dmesg output with initcall_debug
> option enabled and extract some useful information. It can for example be used
> to analyze the output of the grab-boot-data.sh tool that Tim presented on this
> list [1] just a few days ago.
>
> Usage is very simple, as the output of dmesg can be piped directly to it:
>
> dmesg | analyze-initcall-debug.py
>
> If no option is specified, it outputs a brief summary, like the following one
> (obtained on my Beagleplay):
>
> 1758 drivers has been initialized, of which 1758 before userspace
> 119 probes happened outside of the init of their driver
> 0 deferred probes pending
> ---
> Top 10 init/probes durations:
> * 30200000.dss -> 523002us
> * deferred_probe_initcall -> 487483us
> * fd00000.gpu -> 162859us
> * 8000f00.mdio -> 142521us
> * 44043000.system-controller -> 71390us
> * 2-004c -> 71178us
> * 40900000.crypto -> 59350us
> * 8000000.ethernet -> 58419us
> * 44043000.system-controller:clock-controller -> 56599us
> * jent_mod_init -> 52140us
This is odd. On my beagleplay board, jent_mod_init only took 32326us.
(see https://birdcloud.org/boot-time-files/boot-data-timslab-bp1-241203-134136.txt)
I'm assuming that we have similar hardware, but possibly different configs,
kernel cmdlines or kernel versions. I'll take a look at this and see if I can
figure out what the difference is between our systems, that causes the
difference in the duration for this function.
This initcall (jent_mod_init) is on my list of initcalls to research to see if they
can be improved or deferred. I haven't seen any degradation in system
behavior by deferring this initcall on my system, but that could be from
lack of my system doing some related operation that needs the RNG data, or my
own ignorance of the effects of this call.
Can someone comment on what items or operations might depend on jent_mod_init
in early boot? In particular, would we expect any cryptographic problems if
the initcall was deferred to a few seconds after booting?
It can be configured as a module, which makes me think that maybe loading
it sometime in very late boot (or even later) is OK.
jent_mod_init is defined in crypto/jitterentropy-kcapi.c, and controlled by
CONFIG_CRYPTO_JITTERENTROPY (See crypto/Kconfig)
It appears to do some random number generator seeding by measuring
the jitter of a compressiong operation in the kernel. I assume some of the cryptography
configs affect the duration of the operations. The initcall duration results on
my beagleplay seem to be relatively consistent (in the 32ms range consistently),
and from bobs_lab, on machine sam1, the duration range is also consistent
(at between 4800 and 5200 usecs).
I'd be interested to see if the range is consistent between runs on other machines
Francesco - Can you submit another boot-data file (or 2 or 3) for francsecoslab-beagleplay, to
provide some data on this?
Also, can anyone else who sees this initcall in their boot sequence run grab-boot-data.sh
(from https://birdcloud.org/boot-time/Boot-time_Tools)
and submit the data for one or more of their machines?
Stephan Mueller - what are you seeing for the cost of this operation on your machines?
This is one of the first initcalls I'm going to dive into to see if it can be improved or deferred.
Maybe some of the RNG seeding can be held over from a previous boot, in order to
eliminate the overhead on the early portion of a current boot. Any feedback on that
idea?
Thanks.
-- Tim
^ permalink raw reply
* RE: [boot-time] [RFC] analyze-initcall-debug.py - a tool to analyze the initcall debug output
From: Bird, Tim @ 2024-12-03 20:33 UTC (permalink / raw)
To: Francesco Valla, Linux Embedded
In-Reply-To: <1964175.7Z3S40VBb9@fedora.fritz.box>
Hey Francesco,
I appreciate the new tool, and have some feedback below.
I'll separate my response and ideas into 2 categories:
- issues raised from your sample report
- feedback on the tool itself
and send them as separate emails.
> -----Original Message-----
> From: Francesco Valla <francesco@valla.it>
> Dear fellow boot time optimizers,
>
> following the first boot time SIG meeting, which I lurked with much pleasure
> (but didn't participate to, as I was a bit in awe among such authorities), I'd
> like to introduce myself with code rather than a presentation or resume.
>
> Here is a python script which can analyze a dmesg output with initcall_debug
> option enabled and extract some useful information. It can for example be used
> to analyze the output of the grab-boot-data.sh tool that Tim presented on this
> list [1] just a few days ago.
>
> Usage is very simple, as the output of dmesg can be piped directly to it:
>
> dmesg | analyze-initcall-debug.py
>
> If no option is specified, it outputs a brief summary, like the following one
> (obtained on my Beagleplay):
>
> 1758 drivers has been initialized, of which 1758 before userspace
> 119 probes happened outside of the init of their driver
> 0 deferred probes pending
> ---
> Top 10 init/probes durations:
> * 30200000.dss -> 523002us
This call, and a lot of the others are missing function names. Did you compile the kernel with
CONFIG_KALLSYMS=y?
If that's the case, is there a way to use the System.map file for the kernel (used on
the machine where the dmesg was obtained from) to map these addresses
to their respective function names?
> * deferred_probe_initcall -> 487483us
I feel like this function is interfering with the ability to correctly assign
boot time delays to the correct initcall.
I would really like to see someone tackle this issue, by studying the deferred_probe_initcall
function, to see if there's a way to instrument it so that the cost of each
probe can be reported and then associated with corresponding initcall, when initcall_debug
is used.
Does anyone want to look into this issue?
It falls under the category of improving instrumentation and reducing blind spots during
boot.
> * fd00000.gpu -> 162859us
> * 8000f00.mdio -> 142521us
> * 44043000.system-controller -> 71390us
> * 2-004c -> 71178us
> * 40900000.crypto -> 59350us
> * 8000000.ethernet -> 58419us
> * 44043000.system-controller:clock-controller -> 56599us
> * jent_mod_init -> 52140us
>
> A more complete output, reporting all durations as well as statuses, impact of
> probe deferral and so on, can be produced either in HTML or Markdown format,
> using respectively the --html or --markdown parameters.
This seems quite useful, as this will allow me to add this output to my boot-time
wiki quite easily.
>
> Note: it hasn't (yet?) been used in a "production" environment, as I never
> found the motivation to write a script and always analyzed manually the
> initcall_debug output.
>
> If you have time to take a look, please tell me what you think and what
> can be added.
I'll give feedback on the tool itself in a separate message.
-- Tim
>
> [1] https://lore.kernel.org/linux-
> embedded/MW5PR13MB5632F54DCBDA0C74370E531FFD5C2@MW5PR13MB5632.namprd13.prod.outlook.com/
>
^ permalink raw reply
* Grand Piano 12/03
From: Michelle Bukaty @ 2024-12-03 15:27 UTC (permalink / raw)
To: linux-embedded
Hello,
I am reaching out once more about my late husband Yamaha piano, which i am offering to anyone who would genuinely cherish it. If you or someone you know would be interested in receiving this instrument as a gift, please feel free to get in touch.
Warm regards,
Michelle
^ permalink raw reply
* Re: [PATCH v2] printk: Improve memory usage logging during boot
From: Petr Mladek @ 2024-12-03 9:42 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Isaac J. Manjarres, Steven Rostedt, John Ogness,
Sergey Senozhatsky, surenb, kernel-team, linux-kernel,
Linux Embedded
In-Reply-To: <CAMuHMdX_r05iUFw6-Jj8Utry5bHdE6=U46uB4Za1NTsdZOuOMw@mail.gmail.com>
On Sun 2024-12-01 12:40:13, Geert Uytterhoeven wrote:
> Hi Petr et al,
>
> CC linux-embedded
>
> On Tue, Oct 1, 2024 at 5:55 PM Petr Mladek <pmladek@suse.com> wrote:
> > On Mon 2024-09-30 11:48:24, Isaac J. Manjarres wrote:
> > > When the initial printk ring buffer size is updated, setup_log_buf()
> > > allocates a new ring buffer, as well as a set of meta-data structures
> > > for the new ring buffer. The function also emits the new size of the
> > > ring buffer, but not the size of the meta-data structures.
> > >
> > > This makes it difficult to assess how changing the log buffer size
> > > impacts memory usage during boot.
> > >
> > > For instance, increasing the ring buffer size from 512 KB to 1 MB
> > > through the command line yields an increase of 2304 KB in reserved
> > > memory at boot, while the only obvious change is the 512 KB
> > > difference in the ring buffer sizes:
> > >
> > > log_buf_len=512K:
> > >
> > > printk: log_buf_len: 524288 bytes
> > > Memory: ... (... 733252K reserved ...)
> > >
> > > log_buf_len=1M:
> > >
> > > printk: log_buf_len: 1048576 bytes
> > > Memory: ... (... 735556K reserved ...)
> > >
> > > This is because of how the size of the meta-data structures scale with
> > > the size of the ring buffer.
> > >
> > > Even when there aren't changes to the printk ring buffer size (i.e. the
> > > initial size == 1 << CONFIG_LOG_BUF_SHIFT), it is impossible to tell
> > > how much memory is consumed by the printk ring buffer during boot.
> > >
> > > Therefore, unconditionally log the sizes of the printk ring buffer
> > > and its meta-data structures, so that it's easier to understand
> > > how changing the log buffer size (either through the command line or
> > > by changing CONFIG_LOG_BUF_SHIFT) affects boot time memory usage.
> > >
> > > With the new logs, it is much easier to see exactly why the memory
> > > increased by 2304 KB:
> > >
> > > log_buf_len=512K:
> >
> > printk: log buffer data + meta data: 524288 + 1835008 = 2359296 bytes
> >
> > > log_buf_len=1M:
> >
> > printk: log buffer data + meta data: 1048576 + 3670016 = 4718592 bytes
>
> Thanks, this is very useful information!
>
> Isn't this kernel log message bookkeeping becoming a bit too excessive?
> E.g. on a small system:
>
> printk: log buffer data + meta data: 65536 + 204800 = 270336 bytes
>
> Why is the meta data that big (> 3*log buffer)?
>
> #define PRB_AVGBITS 5 /* 32 character average length */
>
> unsigned int descs_count = log_buf_len >> PRB_AVGBITS;
> meta_data_size = descs_count * (sizeof(struct prb_desc) +
> sizeof(struct printk_info));
>
> struct prb_data_blk_lpos {
> unsigned long begin;
> unsigned long next;
> };
>
> struct prb_desc {
> atomic_long_t state_var;
> struct prb_data_blk_lpos text_blk_lpos;
> };
I am afraid that we could not do much about the size of this part.
All the variables are important parts of the lockless machinery.
> i.e. 12 bytes on 32-bit, 24 bytes on 64-bit.
>
> #define PRINTK_INFO_SUBSYSTEM_LEN 16
> #define PRINTK_INFO_DEVICE_LEN 48
>
> struct dev_printk_info {
> char subsystem[PRINTK_INFO_SUBSYSTEM_LEN];
> char device[PRINTK_INFO_DEVICE_LEN];
> };
This is probably the lowest hanging fruit. It should be possible
to write these dev_printk-specific metadata into the data buffer
a more efficient way and only for records created by dev_printk().
It would require adding "dict_len" into "struct printk_info"
and reserving space for both "text_len" and "dict_len".
We bundled it into the metadata because these are metadata.
We wanted to keep the design as clean as possible. We focused
mainly on the stability and maintainability of the code.
And it was really challenging to get it all working.
> struct printk_info {
> u64 seq; /* sequence number */
I do not recal the details. But I think that we need to
explicitely store the 64-bit "seq" number in the metadata
because of the lockless algoritm. It helps to solve
problems with wrapping of the counter in
"atomic_long_t state_var".
It was not stored before. The following global values were
enough when the log buffer was synchronized by "logbuf_lock"
spin lock:
static u64 log_first_seq;
static u64 log_next_seq;
> u64 ts_nsec; /* timestamp in nanoseconds */
> u16 text_len; /* length of text message */
> u8 facility; /* syslog facility */
> u8 flags:5; /* internal record flags */
> u8 level:3; /* syslog level */
> u32 caller_id; /* thread id or processor id */
These metadata used to be stored in the "data" buffer next to the
message. Here is the declaration from v6.9:
struct printk_log {
u64 ts_nsec; /* timestamp in nanoseconds */
u16 len; /* length of entire record */
u16 text_len; /* length of text buffer */
u16 dict_len; /* length of dictionary buffer */
u8 facility; /* syslog facility */
u8 flags:5; /* internal record flags */
u8 level:3; /* syslog level */
#ifdef CONFIG_PRINTK_CALLER
u32 caller_id; /* thread id or processor id */
#endif
}
> struct dev_printk_info dev_info;
As I wrote above. It should be possible to store these metadata more
effectively in the data buffer.
But note that it is only about the internal kernel code. These change
would require also updating external tools, for example "crash" tool.
> };
>
> for a whopping 88 bytes. So that totals to 100 bytes per entry
> on 32-bit, and 112 on 64-bit, for an average of 32 characters per
> printed message...
It would be interesting to know how much if these are wasted because
either struct dev_printk_info is empty or the entries are shorter.
Best Regards,
Petr
^ permalink raw reply
* Re: [PATCH v2] printk: Improve memory usage logging during boot
From: Geert Uytterhoeven @ 2024-12-01 11:40 UTC (permalink / raw)
To: Petr Mladek
Cc: Isaac J. Manjarres, Steven Rostedt, John Ogness,
Sergey Senozhatsky, surenb, kernel-team, linux-kernel,
Linux Embedded
In-Reply-To: <ZvwZV5MDlQYGJv9N@pathway.suse.cz>
Hi Petr et al,
CC linux-embedded
On Tue, Oct 1, 2024 at 5:55 PM Petr Mladek <pmladek@suse.com> wrote:
> On Mon 2024-09-30 11:48:24, Isaac J. Manjarres wrote:
> > When the initial printk ring buffer size is updated, setup_log_buf()
> > allocates a new ring buffer, as well as a set of meta-data structures
> > for the new ring buffer. The function also emits the new size of the
> > ring buffer, but not the size of the meta-data structures.
> >
> > This makes it difficult to assess how changing the log buffer size
> > impacts memory usage during boot.
> >
> > For instance, increasing the ring buffer size from 512 KB to 1 MB
> > through the command line yields an increase of 2304 KB in reserved
> > memory at boot, while the only obvious change is the 512 KB
> > difference in the ring buffer sizes:
> >
> > log_buf_len=512K:
> >
> > printk: log_buf_len: 524288 bytes
> > Memory: ... (... 733252K reserved ...)
> >
> > log_buf_len=1M:
> >
> > printk: log_buf_len: 1048576 bytes
> > Memory: ... (... 735556K reserved ...)
> >
> > This is because of how the size of the meta-data structures scale with
> > the size of the ring buffer.
> >
> > Even when there aren't changes to the printk ring buffer size (i.e. the
> > initial size == 1 << CONFIG_LOG_BUF_SHIFT), it is impossible to tell
> > how much memory is consumed by the printk ring buffer during boot.
> >
> > Therefore, unconditionally log the sizes of the printk ring buffer
> > and its meta-data structures, so that it's easier to understand
> > how changing the log buffer size (either through the command line or
> > by changing CONFIG_LOG_BUF_SHIFT) affects boot time memory usage.
> >
> > With the new logs, it is much easier to see exactly why the memory
> > increased by 2304 KB:
> >
> > log_buf_len=512K:
> >
> > printk: log_buf_len: 524288 bytes
> > printk: prb_descs size: 393216 bytes
> > printk: printk_infos size: 1441792 bytes
>
> This should get updated to the new format.
> If I count correctly then it should be:
>
> printk: log buffer data + meta data: 524288 + 1835008 = 2359296 bytes
>
> > Memory: ... (... 733252K reserved ...)
> >
> > log_buf_len=1M:
> >
> > printk: log_buf_len: 1048576 bytes
> > printk: prb_descs size: 786432 bytes
> > printk: printk_infos size: 2883584 bytes
>
> and here:
>
> printk: log buffer data + meta data: 1048576 + 3670016 = 4718592 bytes
Thanks, this is very useful information!
Isn't this kernel log message bookkeeping becoming a bit too excessive?
E.g. on a small system:
printk: log buffer data + meta data: 65536 + 204800 = 270336 bytes
Why is the meta data that big (> 3*log buffer)?
#define PRB_AVGBITS 5 /* 32 character average length */
unsigned int descs_count = log_buf_len >> PRB_AVGBITS;
meta_data_size = descs_count * (sizeof(struct prb_desc) +
sizeof(struct printk_info));
struct prb_data_blk_lpos {
unsigned long begin;
unsigned long next;
};
struct prb_desc {
atomic_long_t state_var;
struct prb_data_blk_lpos text_blk_lpos;
};
i.e. 12 bytes on 32-bit, 24 bytes on 64-bit.
#define PRINTK_INFO_SUBSYSTEM_LEN 16
#define PRINTK_INFO_DEVICE_LEN 48
struct dev_printk_info {
char subsystem[PRINTK_INFO_SUBSYSTEM_LEN];
char device[PRINTK_INFO_DEVICE_LEN];
};
struct printk_info {
u64 seq; /* sequence number */
u64 ts_nsec; /* timestamp in nanoseconds */
u16 text_len; /* length of text message */
u8 facility; /* syslog facility */
u8 flags:5; /* internal record flags */
u8 level:3; /* syslog level */
u32 caller_id; /* thread id or processor id */
struct dev_printk_info dev_info;
};
for a whopping 88 bytes. So that totals to 100 bytes per entry
on 32-bit, and 112 on 64-bit, for an average of 32 characters per
printed message...
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* [boot-time] [RFC] analyze-initcall-debug.py - a tool to analyze the initcall debug output
From: Francesco Valla @ 2024-11-27 23:35 UTC (permalink / raw)
To: Linux Embedded
Dear fellow boot time optimizers,
following the first boot time SIG meeting, which I lurked with much pleasure
(but didn't participate to, as I was a bit in awe among such authorities), I'd
like to introduce myself with code rather than a presentation or resume.
Here is a python script which can analyze a dmesg output with initcall_debug
option enabled and extract some useful information. It can for example be used
to analyze the output of the grab-boot-data.sh tool that Tim presented on this
list [1] just a few days ago.
Usage is very simple, as the output of dmesg can be piped directly to it:
dmesg | analyze-initcall-debug.py
If no option is specified, it outputs a brief summary, like the following one
(obtained on my Beagleplay):
1758 drivers has been initialized, of which 1758 before userspace
119 probes happened outside of the init of their driver
0 deferred probes pending
---
Top 10 init/probes durations:
* 30200000.dss -> 523002us
* deferred_probe_initcall -> 487483us
* fd00000.gpu -> 162859us
* 8000f00.mdio -> 142521us
* 44043000.system-controller -> 71390us
* 2-004c -> 71178us
* 40900000.crypto -> 59350us
* 8000000.ethernet -> 58419us
* 44043000.system-controller:clock-controller -> 56599us
* jent_mod_init -> 52140us
A more complete output, reporting all durations as well as statuses, impact of
probe deferral and so on, can be produced either in HTML or Markdown format,
using respectively the --html or --markdown parameters.
Note: it hasn't (yet?) been used in a "production" environment, as I never
found the motivation to write a script and always analyzed manually the
initcall_debug output.
If you have time to take a look, please tell me what you think and what
can be added.
Thank you!
Regards,
Francesco
[1] https://lore.kernel.org/linux-embedded/MW5PR13MB5632F54DCBDA0C74370E531FFD5C2@MW5PR13MB5632.namprd13.prod.outlook.com/
--- analyze-initcall-debug.py script follows ---
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
This script can be used to analyze a Linux kernel dmesg output when the
initcall_debug command line output is used. It extract various information,
such as init durations, impact of probe deferral and so on.
"""
import argparse
import re
import sys
ERRCODE_PROBE_DEFER = 517
parser = argparse.ArgumentParser()
parser.add_argument('--dmesg', nargs='?', type=argparse.FileType('r'),
default=sys.stdin)
parser.add_argument('--html', action='store_true')
parser.add_argument('--markdown', action='store_true')
args = parser.parse_args()
class Run:
def __init__(self, start_time:int, end_time:int = -1, duration:int = 0, retval:int = 0):
self._start_time = start_time
self._end_time = end_time
self._duration = duration
self._retval = retval
self._ended = (end_time >= 0)
@property
def start_time(self) -> int:
return self._start_time
@property
def end_time(self) -> int:
return self._end_time
@end_time.setter
def end_time(self, time:int):
self._end_time = time
self._ended = True
@property
def duration(self) -> int:
return self._duration
@duration.setter
def duration(self, time:int):
self._duration = time
@property
def retval(self) -> int:
return self._retval
@retval.setter
def retval(self, val:int):
self._retval = val
@property
def running(self) -> bool:
return (not self._ended or abs(self.retval) == ERRCODE_PROBE_DEFER)
@property
def failed(self) -> bool:
return ((self.retval != 0) and not self.running)
class Entity:
def __init__(self, name:str, start_time:int = 0, end_time:int = 0, duration:int = 0, retval:int = 0):
self._name = name
self._runs = [ Run(start_time, end_time, duration, retval) ]
@property
def name(self) -> str:
return self._name
@property
def first_start_time(self) -> int:
return self._runs[0].start_time
@property
def last_start_time(self) -> int:
return self._runs[-1].start_time
@property
def first_end_time(self) -> int:
return self._runs[0].end_time
@property
def last_end_time(self) -> int:
return self._runs[-1].end_time
@property
def duration(self) -> int:
return sum([ r.duration for r in self._runs ])
@property
def wasted_time(self) -> int:
return sum([ r.duration for r in filter(lambda x: x.failed or abs(x.retval) == ERRCODE_PROBE_DEFER, self._runs) ])
@property
def retval(self) -> int:
return self._runs[-1].retval
@property
def failed(self) -> bool:
return self._runs[-1].failed
@property
def running(self) -> bool:
return self._runs[-1].running
@property
def deferred_probe_pending(self) -> bool:
return (abs(self._runs[-1].retval) == ERRCODE_PROBE_DEFER)
@property
def num_deferred_probes(self) -> int:
return len(list(filter(lambda r: abs(r.retval) == ERRCODE_PROBE_DEFER, self._runs)))
def addStart(self, start_time: int):
self._runs.append( Run(start_time) )
def addEnd(self, end_time:int = 0, duration:int = 0, retval:int = 0):
self._runs[-1].end_time = end_time
self._runs[-1].duration = duration
self._runs[-1].retval = retval
def addRun(self, start_time: int, end_time:int = 0, duration:int = 0, retval:int = 0):
self._runs.append( Run(start_time, end_time, duration, retval) )
class Driver (Entity):
def __init__(self, name: str, start_time: int):
super().__init__(name, start_time)
def addInit(self, init_time: int):
self.addStart(init_time)
def addReturn(self, return_time: int, duration: int, retval: int):
self.addEnd(return_time, duration, retval)
class Probe (Entity):
def __init__(self, name: str, return_time: int, duration: int, retval: int):
super().__init__(name, return_time, return_time, duration, retval)
class Init (Entity):
def __init__(self, name: str, start_time: int):
super().__init__(name, start_time)
# Regex for 'calling' messages
# "[ 0.466115] calling pci_sysfs_init+0x0/0xa8 @ 1"
calling_prog = re.compile(r'\[([0-9\s]+\.[0-9]+)\] calling ([0-9a-zA-Z_]+)\+(0x[0-9a-fA-F]+\/0x[0-9a-fA-F]+) @ ([0-9]+)')
# Regex for 'initcall ... returned' messages
# "[ 0.466115] initcall pci_sysfs_init+0x0/0xa8 returned 0 after 5 usecs"
returned_prog = re.compile(r'\[([0-9\s]+\.[0-9]+)\] initcall ([0-9a-zA-Z_]+)\+(0x[0-9a-fA-F]+\/0x[0-9a-fA-F]+) returned ([\-0-9]+) after ([0-9]+) usecs')
# Regex for 'probe ... returned' messages
# "[ 0.466115] probe of cpufreq-dt returned 517 after 140 usec"
probe_prog = re.compile(r'\[([0-9\s]+\.[0-9]+)\] probe of ([0-9a-zA-Z_\-\.\:@]+) returned ([\-0-9]+) after ([0-9]+) usecs')
# Regex for
# "[ 1.060329] Run /sbin/init as init process"
init_prog = re.compile(r'\[([0-9\s]+\.[0-9]+)\] Run ([/0-9a-zA-Z_]+) as init process')
drivers = list()
probes = list()
init = None
# Extract data from dmesg
for line in args.dmesg:
match = calling_prog.match(line)
if match is not None:
time = float(match.group(1)) * float(10^6)
name = str(match.group(2))
try:
time = float(match.group(1)) * float(10^6)
name = str(match.group(2))
except Exception as e:
print(f'Failed parsing line "{line.rstrip()}" as call')
raise e
else:
for d in filter(lambda x: x.name == name, drivers):
d.addInit(time)
break
else:
drivers.append(Driver(name, time))
finally:
continue
match = returned_prog.match(line)
if match is not None:
try:
time = float(match.group(1)) * float(10^6)
name = str(match.group(2))
retval = int(match.group(4))
duration = int(match.group(5))
except:
print(f'Failed parsing line "{line.rstrip()}" as call return')
else:
for d in filter(lambda x: x.name == name, drivers):
d.addReturn(time, duration, retval)
break
else:
print(f'Detected return for driver {name}, for which an init was never recorded')
finally:
continue
match = probe_prog.match(line)
if match is not None:
try:
time = float(match.group(1)) * float(10^6)
name = str(match.group(2))
retval = int(match.group(3))
duration = int(match.group(4))
except:
print(f'Failed parsing line "{line.rstrip()}" as probe return')
else:
for d in filter(lambda x: x.name == name, probes):
d.addRun(time, time, duration, retval)
break
else:
probes.append(Probe(name, time, duration, retval))
finally:
continue
if init is None:
match = init_prog.match(line)
if match is not None:
try:
time = float(match.group(1)) * float(10^6)
name = str(match.group(2))
except:
print(f'Failed parsing line "{line.rstrip()}" as probe return')
else:
init = Init(name, time)
finally:
continue
if args.markdown:
print('| Name | Type [^1] | Status [^2] | Duration (us) | Before userspace | Num deferred probes | Time spent in failed inits (us) |')
print('| -----| --------- | ----------- | ------------- | ---------------- | ------------------- | ------------------------------- |')
for d in sorted(filter(lambda k: k.duration > 0, drivers + probes), key=lambda k: k.duration, reverse=True):
etype = 'I' if d in drivers else 'P'
run_status = 'D' if d.deferred_probe_pending else 'R' if d.running else f'F({abs(d.retval)})' if d.failed else 'K'
before_init = 'Y' if d.last_start_time < init.last_start_time else 'N'
print(f'| {d.name} | {etype} | {run_status} | {d.duration} | {before_init} | {d.num_deferred_probes} | {d.wasted_time} |')
print('[^1]: I=Init function, P=device Probe')
print('[^2]: D=Deferred probe pending, R=still Running, F(N)=Failed (with errcode N), K=init oK')
elif args.html:
print(
'''
<!DOCTYPE HTML>
<html>
<head>
<meta charset="utf-8" />
<title>Initcall analysis</title>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style>
table, th, td {
border: 1px solid;
}
</style>
</head>
<body>
<table>
<tr>
<th>Name</th>
<th>Type <a href="#note1">[1]</a></th>
<th>Status <a href="#note2">[2]</a></th>
<th>Duration (us)</th>
<th>Before userspace</th>
<th>Num deferred probes</th>
<th>Time spent in failed inits (us)</th>
</tr>''')
for d in sorted(filter(lambda k: k.duration > 0, drivers + probes), key=lambda k: k.duration, reverse=True):
etype = 'I' if d in drivers else 'P'
run_status = 'D' if d.deferred_probe_pending else 'R' if d.running else f'F({abs(d.retval)})' if d.failed else 'K'
before_init = 'Y' if d.last_start_time < init.last_start_time else 'N'
print(
f'''
<tr>
<td>{d.name}</td>
<td>{etype}</td>
<td>{run_status}</td>
<td>{d.duration}</td>
<td>{before_init}</td>
<td>{d.num_deferred_probes}</td>
<td>{d.wasted_time}</td>
</tr>''')
print(
'''
</table>
<p id="note1">[1] I=Init function, P=device Probe</p>
<p id="note2">[2] D=Deferred probe pending, R=still Running, F(N)=Failed (with errcode N), K=init oK</p>
</body>
</html>
''')
else:
print(f'{len(drivers)} drivers has been initialized, of which {len(list(filter(lambda d: d.last_start_time < init.last_start_time, drivers)))} before userspace')
print(f'{len(probes)} probes happened outside of the init of their driver')
print(f'{len(list(filter(lambda d: d.deferred_probe_pending, drivers + probes)))} deferred probes pending')
print('---')
print('Top 10 init/probes durations:')
for d in sorted(filter(lambda k: k.duration > 0, drivers + probes), key=lambda k: k.duration, reverse=True)[0:10]:
print(f' * {d.name} -> {d.duration}us')
^ permalink raw reply
* RE: [boot-time] RFC grab-boot-data.sh - tool to grab boot time data
From: Bird, Tim @ 2024-11-27 19:20 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: linux-embedded@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <CAMuHMdWUMap1maZCmOqEqaxNWHKRiwg1QaN0W9j+8iPkPyHEMg@mail.gmail.com>
> -----Original Message-----
> From: Geert Uytterhoeven <geert@linux-m68k.org>
> Hi Tim,
>
> On Thu, Nov 7, 2024 at 2:26 AM Bird, Tim <Tim.Bird@sony.com> wrote:
> > I'm writing to introduce a tool that I'd like people to run, to gather boot-time data on
> > their systems. I'd like to collect boot-time data from a lot of different systems.
>
> [...]
>
> Thank you!
>
> > ---- script grab-boot-data.sh follows ----
> > #!/bin/sh
> > # grab-boot-data.sh
>
> It's always a good idea to run shellcheck on any script you write.
Thanks. I went through the issues reported by shellcheck and adjusted
a few things.
The tool currently lives at: https://birdcloud.org/boot-time/Boot-time_Tools
but I anticipate creating, soon, a github repo for this and other tools, and possibly
non-upstreamable (or on-their-way-to-upstream) patches.
-- Tim
^ permalink raw reply
* (no subject)
From: Marko Hoyer @ 2024-11-24 21:27 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <99a4b033-4459-467b-9ccd-12c6188e7977@freenet.de>
|subscribe linux-embedded|
^ permalink raw reply
* RE: [boot-time] SIG organizing meeting
From: Bird, Tim @ 2024-11-23 23:37 UTC (permalink / raw)
To: Bird, Tim, linux-embedded@vger.kernel.org
In-Reply-To: <MW5PR13MB5632E7C5AEDF8AAC82EF8192FD2C2@MW5PR13MB5632.namprd13.prod.outlook.com>
I swear I proofread this multiple times. This video/conference call is
planned for Tuesday, November 26th.
> -----Original Message-----
> From: Bird, Tim <Tim.Bird@sony.com>
> > -----Original Message-----
> > From: Bird, Tim
> > It's about time to plan our first meeting of the Linux boot-time SIG (Special Interest Group).
>
> Hey Linux Boot-Time SIG interested parties,
>
> Here is the information for the first Linux Boot-Time SIG conference call.
>
> The meeting will be held via the Jitsi online meeting platform.
> Here is the connection/call-in information:
> ----
> To Join the meeting via web, click on:
> https://meet.jit.si/LinuxBootTimeSIG
>
> To join by phone instead (in the U.S.), tap this: +1.512.647.1431,,3337584763#
>
> Looking for a different dial-in number?
> See meeting dial-in numbers: https://meet.jit.si/static/dialInInfo.html?room=LinuxBootTimeSIG
>
> If also dialing-in through a room phone, join without connecting to audio: https://meet.jit.si/LinuxBootTimeSIG#config.startSilent=true
> ----
> I will start the meeting at 9:00 am Mountain Time, see this link for other time zones:
> https://www.timeanddate.com/worldclock/meetingdetails.html?year=2024&month=11&day=26&hour=16&min=0&sec=0&p1=220&p2=
> 137&p3=195&p4=771
>
> (That makes it 8:00 am Pacific, 16:00 UTC, 17:00 CET, and 21:30 IST)
>
> We'll plan for a 1-hour meeting.
>
> The initial agenda will be:
> - governance of the SIG
> - projects to pursue?
> - frequency of meetings?
> - what resources to use?
> - where to put things (besides (obviously) upstream)?
> - how to collaborate?
>
> The meeting is open to anyone.
>
> I look forward to talking with you then.
> -- Tim
>
>
>
^ permalink raw reply
* RE: [boot-time] SIG organizing meeting
From: Bird, Tim @ 2024-11-23 23:35 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org
In-Reply-To: <MW5PR13MB5632D59CC8D54682BE6EA092FD5D2@MW5PR13MB5632.namprd13.prod.outlook.com>
> -----Original Message-----
> From: Bird, Tim
> It's about time to plan our first meeting of the Linux boot-time SIG (Special Interest Group).
Hey Linux Boot-Time SIG interested parties,
Here is the information for the first Linux Boot-Time SIG conference call.
The meeting will be held via the Jitsi online meeting platform.
Here is the connection/call-in information:
----
To Join the meeting via web, click on:
https://meet.jit.si/LinuxBootTimeSIG
To join by phone instead (in the U.S.), tap this: +1.512.647.1431,,3337584763#
Looking for a different dial-in number?
See meeting dial-in numbers: https://meet.jit.si/static/dialInInfo.html?room=LinuxBootTimeSIG
If also dialing-in through a room phone, join without connecting to audio: https://meet.jit.si/LinuxBootTimeSIG#config.startSilent=true
----
I will start the meeting at 9:00 am Mountain Time, see this link for other time zones:
https://www.timeanddate.com/worldclock/meetingdetails.html?year=2024&month=11&day=26&hour=16&min=0&sec=0&p1=220&p2=137&p3=195&p4=771
(That makes it 8:00 am Pacific, 16:00 UTC, 17:00 CET, and 21:30 IST)
We'll plan for a 1-hour meeting.
The initial agenda will be:
- governance of the SIG
- projects to pursue?
- frequency of meetings?
- what resources to use?
- where to put things (besides (obviously) upstream)?
- how to collaborate?
The meeting is open to anyone.
I look forward to talking with you then.
-- Tim
^ permalink raw reply
* [boot-time] SIG organizing meeting
From: Bird, Tim @ 2024-11-08 23:47 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org; +Cc: linux-kernel@vger.kernel.org
Hello fellow embedded Linux developers,
It's about time to plan our first meeting of the Linux boot-time SIG (Special Interest Group).
Things have been very loosely organized as I've been trying to gauge interest
in the SIG, and put forth some ideas around Linux boot-time to this list.
Unfortunately, I'll be traveling the next few weeks, so the earliest I can put something
together is the week of November 25th. I appears that people who have contacted
me with interest in the SIG are primarily from the US and Europe, and maybe India.
(It's hard to tell just from e-mails). Therefore I propose some kind of video-conference
at 9:00 am Mountain Time (I'm in that time zone) on Tuesday, November 26th.
That makes it 8:00 am Pacific, 5:00 pm CET, and 9:30 pm IST. Not great, but possibly OK.
The agenda would include a number of things, including:
- governance of the SIG
- projects to pursue
- frequency of meetings
- what resources to use
- where to put things (besides (obviously) upstream)
- how to collaborate, etc.
This is not a traditional trade association or funded project startup, so there
won't be any memberships (with fees, tiers, voting rights, etc.). But I still
think there should be some designated leadership and resources that will
be available for the SIG to use.
If I had my way, I'd use one of the Plumbers hack rooms for the meeting,
but there are details for access to that, that I need to work out. If anyone
can volunteer to provide the video conference hosting for the first SIG meeting,
please contact me privately. (Otherwise, it might be Teams, which is what I have.)
Please let me know any feedback you have on these plans. If you have a strong
idea that you would like to attend the meeting, you can ping me privately to
let me know your interest. (Fair warning - I'll be looking for volunteers for some
tasks, so if you are easily talked into Open Source work, consider your
attendance carefully. That said, lurkers are welcome also.)
A few other thoughts...
I've been a little wary about how much to copy LKML on these messages.
Usually, most of the kernel mailing lists are highly focused on technical
discussions and patch reviews. This SIG management stuff is neither of
those, and I don't want to unduly burden the kernel lists with this type
of material. To that end, I will probably stop CC-ing LKML on this type
of administrative e-mail. Also, it may be appropriate to get a separate
email list or communication channel for SIG-related conversations (ie
discussions unrelated to kernel features.) Let me know what you think.
Regards,
-- Tim
^ permalink raw reply
* Re: Request to join the mailing list
From: Ezra Buehler @ 2024-11-08 15:20 UTC (permalink / raw)
To: Madeeha Javed; +Cc: linux-embedded
In-Reply-To: <CAKH5gYwJL85NYma7YaiFwN8Bz-WngHa=arvR-SfKtr_FFD3pSw@mail.gmail.com>
Hi Madeeha,
On Fri, Nov 8, 2024 at 3:35 PM Madeeha Javed <madeeha@gmail.com> wrote:
> I work at Igalia but I am joining from my personal email account
> because I have personal interest in working on Linux boot time
> optimizations due to my prior experience with Automotive use-cases in
> my past job.
>
> Thanks!
> Madeeha Javed
> https://www.linkedin.com/in/madeehajaved/
>
On how to subscribe, from https://elinux.org/Boot_Time#Mailing_list:
To subscribe or unsubscribe, see this page:
https://subspace.kernel.org/vger.kernel.org.html
Note: don't send email to the list itself to subscribe or unsubscribe.
Click on the 'sub' or 'unsub' buttons on the web page, or follow the
instructions here: https://subspace.kernel.org/subscribing.html
Cheers,
Ezra.
^ permalink raw reply
* Request to join the mailing list
From: Madeeha Javed @ 2024-11-08 14:35 UTC (permalink / raw)
To: linux-embedded
I work at Igalia but I am joining from my personal email account
because I have personal interest in working on Linux boot time
optimizations due to my prior experience with Automotive use-cases in
my past job.
Thanks!
Madeeha Javed
https://www.linkedin.com/in/madeehajaved/
^ permalink raw reply
* RE: Boot-time initiative (SIG) thoughts and next steps
From: Bird, Tim @ 2024-11-07 19:35 UTC (permalink / raw)
To: Saravana Kannan
Cc: Brian Masney, linux-embedded@vger.kernel.org,
linux-kernel@vger.kernel.org
In-Reply-To: <CAGETcx-5r6+-9q1GCOry6PP1hALMuPbwSco7jdcNZ24MTMcBpw@mail.gmail.com>
> -----Original Message-----
> From: Saravana Kannan <saravanak@google.com>
> On Tue, Nov 5, 2024 at 4:45 PM Bird, Tim <Tim.Bird@sony.com> wrote:
[snip]
> > We should discuss what a kernel boot-time tuning guide should look like
> > and how it should be organized. Starting this may be one of the first
> > things the SIG does.
>
> I wrote something like this for the Android ecosystem
> https://source.android.com/docs/core/architecture/kernel/boot-time-opt
I just took a look at this and it looks like a very handy document.
I think having something like this in the upstream kernel doc would
be great.
> So, I'm happy to add something like this to the kernel doc if we have
> some consensus on having a kernel doc.
I don't hear any objections. I guess we'll get some feedback when we
post some patches to the linux-doc mailing list.
I would say go for it, and copy this list so we can discuss the planned
document format (and content, of course).
A lot of the information in the android doc would apply to the kernel doc.
-- Tim
^ permalink raw reply
* Re: [boot-time] Please check this wiki page about RCU expedited mode
From: Eric Curtin @ 2024-11-07 11:48 UTC (permalink / raw)
To: Bird, Tim
Cc: linux-embedded@vger.kernel.org, edchong@redhat.com, Brian Masney,
linux-kernel@vger.kernel.org
In-Reply-To: <MW5PR13MB563287E9DCEF5F43C0B2BB0AFD5C2@MW5PR13MB5632.namprd13.prod.outlook.com>
It looks great to me, it's probably kinda self-explanatory but we
could put that it's only relevant to RT-kernels here:
Kernel Version put kernel version here
Configuration list relevant kernel config options here
Is mise le meas/Regards,
Eric Curtin
On Thu, 7 Nov 2024 at 01:50, Bird, Tim <Tim.Bird@sony.com> wrote:
>
> Ed, Brian or Eric,
>
> Can you please take a look at this wiki page, and correct
> mistakes or add any material you think would be valuable.
>
> https://elinux.org/RCU_expedited_mode
>
> If you have a specific example of a boot time reduction
> from using this technique, can you add it to the case
> study listed at the bottom of the page.
>
> I copied some material from your devconf.us presentation.
>
> If you don't have elinux wiki accounts, you can just reply
> to this email with suggestions, and I'll change or add the material myself.
> One specific question I have is whether this technique only applies
> to an RT kernel or not.
>
> IMHO we can collect information on various techniques on the wiki
> (including data about their savings on different systems and in
> different scenarios), and then migrate the material to an upstream
> kernel tuning guide when we've establish the general utility
> of the material.
>
> As an aside question - if it would be easier to just review the material
> on-list, let me know and I'll send the raw page (in mediawiki markup)
> to the list for email-style review.
>
> Thanks
> -- Tim
>
^ permalink raw reply
* Re: [boot-time] RFC grab-boot-data.sh - tool to grab boot time data
From: Geert Uytterhoeven @ 2024-11-07 7:41 UTC (permalink / raw)
To: Bird, Tim; +Cc: linux-embedded@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <MW5PR13MB5632F54DCBDA0C74370E531FFD5C2@MW5PR13MB5632.namprd13.prod.outlook.com>
Hi Tim,
On Thu, Nov 7, 2024 at 2:26 AM Bird, Tim <Tim.Bird@sony.com> wrote:
> I'm writing to introduce a tool that I'd like people to run, to gather boot-time data on
> their systems. I'd like to collect boot-time data from a lot of different systems.
[...]
Thank you!
> ---- script grab-boot-data.sh follows ----
> #!/bin/sh
> # grab-boot-data.sh
It's always a good idea to run shellcheck on any script you write.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* [boot-time] Please check this wiki page about RCU expedited mode
From: Bird, Tim @ 2024-11-07 1:49 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org, edchong@redhat.com, Brian Masney,
Eric Curtin
Cc: linux-kernel@vger.kernel.org
Ed, Brian or Eric,
Can you please take a look at this wiki page, and correct
mistakes or add any material you think would be valuable.
https://elinux.org/RCU_expedited_mode
If you have a specific example of a boot time reduction
from using this technique, can you add it to the case
study listed at the bottom of the page.
I copied some material from your devconf.us presentation.
If you don't have elinux wiki accounts, you can just reply
to this email with suggestions, and I'll change or add the material myself.
One specific question I have is whether this technique only applies
to an RT kernel or not.
IMHO we can collect information on various techniques on the wiki
(including data about their savings on different systems and in
different scenarios), and then migrate the material to an upstream
kernel tuning guide when we've establish the general utility
of the material.
As an aside question - if it would be easier to just review the material
on-list, let me know and I'll send the raw page (in mediawiki markup)
to the list for email-style review.
Thanks
-- Tim
^ permalink raw reply
* [boot-time] RFC grab-boot-data.sh - tool to grab boot time data
From: Bird, Tim @ 2024-11-07 1:26 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org; +Cc: linux-kernel@vger.kernel.org
Fellow boot time developers,
I'm writing to introduce a tool that I'd like people to run, to gather boot-time data on
their systems. I'd like to collect boot-time data from a lot of different systems.
There are multiple reasons I'd like to collect this data, many of which are explained
on this wiki page:
https://birdcloud.org/boot-time/Boot-time_Tools
The script uploads the data to my personal machine (birdcloud.org), and will
be accessible to the public on this page:
https://birdcloud.org/boot-time/Boot_Data
Please NOTE that this script automatically (unless overridden) sends the data to
one of my personal wikis, on birdcloud.org. This is kind of a poor man's kcidb,
expressly intended to collect boot time data for off-DUT analysis.
(DUT = Device Under Test). You can examine the script to see the operations
it performs, which include things like grabbing your kernel config, cat'ing
the running processes, getting your kernel boot messages, and showing memory,
uptime, processors, mounts and other machine info and kernel info.
If any of this might contain or reveal sensitive information (e.g. the name of a
not-yet-released product, program or peripheral, or the attributes or capabilities thereof),
then you should NOT run this script.
It is possible to run the script while suppressing the upload functionality, using
the -x option. In that case, you can examine the actual data inside the boot-data
file, and determine if you think it contains any sensitive information before sending it.
There are multiple goals for collecting this data, including:
* identifying printk messages (and thus boot operations) that are common to all systems
* collecting data about "common" timings for kernel operations and initcalls
* correlating operation timings with machine attributes (e.g. kernel version, cpu count,
cpu speed, memory size, kernel command line args, selected drivers, etc.)
* detecting kernel operations that run before the clock is initialized on a system
Each of these goals is described in more detail on the ..Tools wiki page listed above.
Please note that this personal wiki is meant solely as a repository of boot data from this
one tool. I plan to add some analysis tools and possibly visualization pages for
the data in the coming months. It is NOT where I plan to put the boot-time
pages related to tools, techniques, projects, etc for the boot-time SIG. That is still
planned to be hosted on the elinux wiki.
The script text appears below, but you can also download a copy from the ..Tools wiki
page listed above.
== Call for action ==
Please run this script on a system (or multiple systems), so that I can collect data
for this effort.
Please also provide feedback on the script, the goals, and this overall approach,
by responding to this email. Please let me know what you think.
Thanks,
-- Tim
---- script grab-boot-data.sh follows ----
#!/bin/sh
# grab-boot-data.sh
# This is a simple script to grab boot time data from dmesg
# gather meta-data about the machine, so the effects of things like
# memory, cpu count, cpu frequency, bogomips, kernel command-line options,
# etc. can be correlated with the duration of specific boot operations
VERSION="1.0"
# for testing
#UPLOAD_URL="http://localhost:8000/cgi-bin/tbwiki.cgi/Boot_Data?action=do_upload"
UPLOAD_URL="https://birdcloud.org/boot-time/Boot_Data?action=do_upload"
usage() {
cat <<HERE
Usage: grab-boot-data.sh -l <lab> -m <machine> [options]
Collect machine information and boot data, and send it to the
birdcloud boot-time wiki, for analysis.
Normally, you run this soon after booting your machine. You should
add the following kernel command line options to your boot configuration:
quiet
initcall_debug
log_buf_len=10M
You may need to add these to a booloader configuration file, such
as grub.cfg (if using the 'grub' bootloader, or if using u-boot,
by editing the bootargs variable). See bootloader-specific documentation
for how to adjust the kernel command line for your boot.
Options:
-h,--help Show this online usage help
-s Skip kernel command-line checks
Use this if you want to collect machine data and dmesg data,
but didn't use 'quiet' or 'initcall_debug' on the kernel
command line.
-l <lab> Specify lab (or user) name for data
-m <name> Specify machine name for data
-d <dir> Output the boot-data file to the indicated directory
-x Don't upload the data to the wiki (but still save the data
to a file)
-u <file> Upload specified boot-data file to the wiki
This should be used when the target doesn't have networking or
is missing the curl command. In this case, use '-x' when
running on the target machine, transfer the resulting
data file to a machine that does have networking and curl,
and then upload the file using -u.
--version Show program version information
--debug Show debug information while running
HERE
}
timestamp() {
date +"%y%m%d-%H%M%S"
}
OUTPUT_DIR=.
LAB="unknown"
MACHINE="unknown"
SAVE_ARGS="$@"
# parse options
while [ -n "$1" ] ; do
case $1 in
-h|--help)
usage
exit 0
;;
-s)
echo "Skipping command line checks"
SKIP_CMDLINE_CHECKS=1
shift
;;
-x)
echo "Skipping upload"
SKIP_UPLOAD=1
shift
;;
-d)
shift
OUTPUT_DIR="$1"
shift
;;
-l)
shift
LAB="$1"
shift
;;
-m)
shift
MACHINE="$1"
shift
;;
-u)
shift
SKIP_CMDLINE_CHECKS=1
SKIP_GRAB=1
UPLOAD_FILE=$1
shift
;;
--version)
echo "grab-boot-data.sh Version $VERSION"
exit 0
;;
--debug)
set -x
shift
;;
*)
echo "Unrecognized command line option '$1'"
echo "Use -h for help"
exit 1
;;
esac
done
OUTFILE="boot-data-${LAB}-${MACHINE}-$(timestamp).txt"
OUTPATH=${OUTPUT_DIR}/${OUTFILE}
# do some error checking
if [ -z "$SKIP_GRAB" -a "$LAB" = "unknown" ] ; then
echo "Error: Please specify a lab name for the boot data"
exit 1
fi
if [ -z "$SKIP_GRAB" -a "$MACHINE" = "unknown" ] ; then
echo "Error: Please specify a machine name for the boot data"
exit 1
fi
# check if 'quiet' and 'initcall_debug' are in the kernel command line
CMDLINE="$(cat /proc/cmdline)"
if [ -z "$SKIP_CMDLINE_CHECKS" -a "${CMDLINE#*quiet}" = "${CMDLINE}" ] ; then
echo "Error: Missing 'quiet' on kernel command line"
echo "Please reboot the kernel with 'quiet' on the command line, and run again"
exit 1
fi
if [ -z "$SKIP_CMDLINE_CHECKS" -a "${CMDLINE#*initcall_debug}" = "${CMDLINE}" ] ; then
echo "Error: Missing 'initcall_debug' on kernel command line"
echo "Please reboot the kernel with 'initcall_debug' on the command line, and run again"
exit 1
fi
out_section() {
echo "== $1 ==" >>$OUTPATH
$2 >>$OUTPATH 2>&1
echo >>$OUTPATH
}
get_config() {
# check a few different places for the kernel config file
if [ -f /proc/config.gz ] ; then
zcat /proc/config.gz
return
fi
# check /lib/modules
RELEASE="$(uname -r)"
if [ -f /lib/modules/${RELEASE}/kernel/kernel/configs.ko ] ; then
insmod /lib/modules/${RELEASE}/kernel/kernel/configs.ko
zcat /proc/config.gz
rmmod configs
return
fi
if [ -f /lib/modules/${RELEASE}/kernel/kernel/configs.ko.xz ] ; then
insmod /lib/modules/${RELEASE}/kernel/kernel/configs.ko.xz
zcat /proc/config.gz
rmmod configs
return
fi
if [ -f /lib/modules/${RELEASE}/build/.config ] ; then
cat /lib/modules/${RELEASE}/build/.config
return
fi
# check /boot directory
if [ -f /boot/config-${RELEASE} ] ; then
cat /boot/config-${RELEASE}
return
fi
if [ -f /boot/config ] ; then
cat /boot/config
return
fi
# other possibilities, though these are less likely on embedded devices
# IMHO - it's too easy for these to be inaccurate, comment them
# out for now
# /usr/src/linux-$RELEASE/.config
# /usr/src/linux/.config
# /usr/lib/ostree-boot/config-$RELEASE
# /usr/lib/kernel/config-$RELEASE
# /usr/src/linux-headers-$RELEASE/.config",
echo "Can't find kernel config data or file"
}
if [ -z "$SKIP_GRAB" ] ; then
echo "=== Machine Info ==============================" >$OUTPATH
out_section UPTIME "uptime"
echo "== GRAB-BOOT-DATA INFO ==" >>$OUTPATH
echo "GBD_ARGS=\"$SAVE_ARGS\"" >>$OUTPATH
echo "GBD_VERSION=\"$VERSION\"" >>$OUTPATH
echo >>$OUTPATH
echo "== Kernel Info ==" >>$OUTPATH
echo "KERNEL_VERSION=\"$(uname -r -v)\"" >>$OUTPATH
echo "KERNEL_CMDLINE=\"$CMDLINE\"" >>$OUTPATH
echo >>$OUTPATH
out_section OS "cat /etc/os-release"
out_section MEMORY "free"
out_section "DISK USAGE" "df -h"
out_section MOUNTS "mount"
# ps -A seems to work with procps ps, busybox ps, and toybox ps
out_section PROCESSES "ps -A"
#out_section INTERRUPTS "cat /proc/interrupts"
out_section CORES "cat /proc/cpuinfo"
out_section CONFIG "get_config"
out_section "KERNEL MESSAGES" "dmesg"
echo >>$OUTPATH
echo "Boot data is in the file: $OUTPATH"
UPLOAD_FILE="${OUTPATH}"
fi
# should I also get a systemd-analyze here?
### upload the data
if [ -z "$SKIP_UPLOAD" ] ; then
echo "Upload file=$UPLOAD_FILE"
RESULT_HTML_FILE="$(mktemp)"
echo "Uploading data ..."
curl -F submit_button=Upload -F file_1=@$UPLOAD_FILE -F file_2= -F file_3= $UPLOAD_URL -o $RESULT_HTML_FILE
if grep "uploaded successfully" $RESULT_HTML_FILE >/dev/null 2>&1 ; then
echo "Data uploaded successfully."
rm $RESULT_HTML_FILE
else
echo "Error: There was a problem uploading the data"
echo "See $RESULT_HTML_FILE for details."
echo "Don't forget to remove this file when done using it."
fi
fi
^ permalink raw reply
* Re: Boot-time initiative (SIG) thoughts and next steps
From: Saravana Kannan @ 2024-11-06 3:41 UTC (permalink / raw)
To: Bird, Tim
Cc: Brian Masney, linux-embedded@vger.kernel.org,
linux-kernel@vger.kernel.org
In-Reply-To: <MW5PR13MB5632276F33CCE4E18D0258E0FD532@MW5PR13MB5632.namprd13.prod.outlook.com>
On Tue, Nov 5, 2024 at 4:45 PM Bird, Tim <Tim.Bird@sony.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Brian Masney <bmasney@redhat.com>
> > Sent: Tuesday, November 5, 2024 5:10 PM
> > To: Saravana Kannan <saravanak@google.com>
> > Cc: Bird, Tim <Tim.Bird@sony.com>; linux-embedded@vger.kernel.org; linux-kernel@vger.kernel.org
> > Subject: Re: Boot-time initiative (SIG) thoughts and next steps
> >
> > On Mon, Oct 28, 2024 at 03: 33: 29PM -0700, Saravana Kannan wrote: > On Sun, Oct 27, 2024 at 6: 30 PM Bird, Tim
> > <Tim. Bird@ sony. com> wrote: > > > On Fri, Oct 25, 2024 at 11: 18 AM Bird, Tim <Tim. Bird@ sony. com> wrote: >
> >
> > On Mon, Oct 28, 2024 at 03:33:29PM -0700, Saravana Kannan wrote:
> > > On Sun, Oct 27, 2024 at 6:30 PM Bird, Tim <Tim.Bird@sony.com> wrote:
> > > > > On Fri, Oct 25, 2024 at 11:18 AM Bird, Tim <Tim.Bird@sony.com> wrote:
> > > > > > = wiki account =
> > > > > > The wiki where we'll be maintaining information about
> > > > > > boot time, and about activities of the boot time SIG, is the elinux wiki.
> > > > > > The page we'll be focusing on is: https://elinux.org/Boot_Time.
> > > > > > If you are interested in helping update and maintain the information there
> > > > > > (which I hope almost everyone is), then please make sure you have a user
> > > > > > account on the wiki.
> > > > > > If you don't have one, please go here:
> > > > > > https://elinux.org/Special:RequestAccount
> > > > > > I have to manually approve accounts in order to fight spambots. It might
> > > > > > take a few days for me to get to your request. It's very helpful if you
> > > > > > put a comment in one of the request fields about this being related to
> > > > > > the boot-time initiative or SIG, so I can distinguish your request from
> > > > > > spam requests.
> > > > >
> > > > > Can we instead keep this all a part of the kernel docs instead of the
> > > > > wiki? Couple of reasons for that:
> > > >
> > > > Ideally, we would put some material in the wiki, and also
> > > > produce a document - some kind of "boot-time tuning guide" that can
> > > > live in the kernel tree.
> > >
> > > This is the part I care most about being in the kernel docs. Eg: what
> > > configs to use. What commandline params to set. Dos and Don'ts for the
> > > drivers, etc. So, good to see that is an acceptable option.
> >
> > I'm interested to help contribute to a boot speed document, and I
> > suspect some others at Red Hat are interested as well. Personally,
> > I would prefer to have a section in the kernel documentation over a
> > Wiki
> OK - that's at least two votes for an upstream kernel doc.
>
> I think it would be good to have a boot time tuning guide upstream.
> I hope to augment that with a tool that will check for various
> items or conditions on a machine, and give a list of recommendations.
> This could go along with the tuning guide.
>
> However, I would still plan to collect some information on the wiki
> that I don't think will be upstreamable. For example, see this page:
> https://elinux.org/Disable_Console
>
> The first few items on that page could be sentences in a kernel tuning
> guide, but the data from actual uses I don't think belongs there, as
> it will quickly bitrot. (Indeed the information on that page has already
> bitrotted.)
>
> I think it would be useful to gather this results information on a wiki, but its
> possible that if reports are sent by e-mail, just a few lore links would
> suffice.
>
> > Besides arch-specific recommendations, we can also contribute
> > some boot speed improvement techniques that we've done that are
> > specific to RT.
> That would be great.
>
> > In addition to the recommended configs, I think it would also be
> > beneficial to list some upstream patches that improve boot speed along
> > with the kernel version it was introduced in.
> I think this would be good also.
>
> We should discuss what a kernel boot-time tuning guide should look like
> and how it should be organized. Starting this may be one of the first
> things the SIG does.
I wrote something like this for the Android ecosystem
https://source.android.com/docs/core/architecture/kernel/boot-time-opt
So, I'm happy to add something like this to the kernel doc if we have
some consensus on having a kernel doc.
-Saravana
> The other one may be a data-gathering tool that
> I've been working on, that (of all things) automatically populates a wiki
> with boot time data. I will explain my intentions when I post the script
> (likely tomorrow).
> -- Tim
>
^ permalink raw reply
* RE: Boot-time initiative (SIG) thoughts and next steps
From: Bird, Tim @ 2024-11-06 0:45 UTC (permalink / raw)
To: Brian Masney, Saravana Kannan
Cc: linux-embedded@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <Zyqz1LBDXZosrjle@x1>
> -----Original Message-----
> From: Brian Masney <bmasney@redhat.com>
> Sent: Tuesday, November 5, 2024 5:10 PM
> To: Saravana Kannan <saravanak@google.com>
> Cc: Bird, Tim <Tim.Bird@sony.com>; linux-embedded@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: Re: Boot-time initiative (SIG) thoughts and next steps
>
> On Mon, Oct 28, 2024 at 03: 33: 29PM -0700, Saravana Kannan wrote: > On Sun, Oct 27, 2024 at 6: 30 PM Bird, Tim
> <Tim. Bird@ sony. com> wrote: > > > On Fri, Oct 25, 2024 at 11: 18 AM Bird, Tim <Tim. Bird@ sony. com> wrote: >
>
> On Mon, Oct 28, 2024 at 03:33:29PM -0700, Saravana Kannan wrote:
> > On Sun, Oct 27, 2024 at 6:30 PM Bird, Tim <Tim.Bird@sony.com> wrote:
> > > > On Fri, Oct 25, 2024 at 11:18 AM Bird, Tim <Tim.Bird@sony.com> wrote:
> > > > > = wiki account =
> > > > > The wiki where we'll be maintaining information about
> > > > > boot time, and about activities of the boot time SIG, is the elinux wiki.
> > > > > The page we'll be focusing on is: https://elinux.org/Boot_Time.
> > > > > If you are interested in helping update and maintain the information there
> > > > > (which I hope almost everyone is), then please make sure you have a user
> > > > > account on the wiki.
> > > > > If you don't have one, please go here:
> > > > > https://elinux.org/Special:RequestAccount
> > > > > I have to manually approve accounts in order to fight spambots. It might
> > > > > take a few days for me to get to your request. It's very helpful if you
> > > > > put a comment in one of the request fields about this being related to
> > > > > the boot-time initiative or SIG, so I can distinguish your request from
> > > > > spam requests.
> > > >
> > > > Can we instead keep this all a part of the kernel docs instead of the
> > > > wiki? Couple of reasons for that:
> > >
> > > Ideally, we would put some material in the wiki, and also
> > > produce a document - some kind of "boot-time tuning guide" that can
> > > live in the kernel tree.
> >
> > This is the part I care most about being in the kernel docs. Eg: what
> > configs to use. What commandline params to set. Dos and Don'ts for the
> > drivers, etc. So, good to see that is an acceptable option.
>
> I'm interested to help contribute to a boot speed document, and I
> suspect some others at Red Hat are interested as well. Personally,
> I would prefer to have a section in the kernel documentation over a
> Wiki
OK - that's at least two votes for an upstream kernel doc.
I think it would be good to have a boot time tuning guide upstream.
I hope to augment that with a tool that will check for various
items or conditions on a machine, and give a list of recommendations.
This could go along with the tuning guide.
However, I would still plan to collect some information on the wiki
that I don't think will be upstreamable. For example, see this page:
https://elinux.org/Disable_Console
The first few items on that page could be sentences in a kernel tuning
guide, but the data from actual uses I don't think belongs there, as
it will quickly bitrot. (Indeed the information on that page has already
bitrotted.)
I think it would be useful to gather this results information on a wiki, but its
possible that if reports are sent by e-mail, just a few lore links would
suffice.
> Besides arch-specific recommendations, we can also contribute
> some boot speed improvement techniques that we've done that are
> specific to RT.
That would be great.
> In addition to the recommended configs, I think it would also be
> beneficial to list some upstream patches that improve boot speed along
> with the kernel version it was introduced in.
I think this would be good also.
We should discuss what a kernel boot-time tuning guide should look like
and how it should be organized. Starting this may be one of the first
things the SIG does. The other one may be a data-gathering tool that
I've been working on, that (of all things) automatically populates a wiki
with boot time data. I will explain my intentions when I post the script
(likely tomorrow).
-- Tim
^ permalink raw reply
* Re: Boot-time initiative (SIG) thoughts and next steps
From: Brian Masney @ 2024-11-06 0:09 UTC (permalink / raw)
To: Saravana Kannan
Cc: Bird, Tim, linux-embedded@vger.kernel.org,
linux-kernel@vger.kernel.org
In-Reply-To: <CAGETcx-Y6LHpZZUeexeuSF4RJ1E2MDtNtST=ytEUPAj7kKzwFA@mail.gmail.com>
On Mon, Oct 28, 2024 at 03:33:29PM -0700, Saravana Kannan wrote:
> On Sun, Oct 27, 2024 at 6:30 PM Bird, Tim <Tim.Bird@sony.com> wrote:
> > > On Fri, Oct 25, 2024 at 11:18 AM Bird, Tim <Tim.Bird@sony.com> wrote:
> > > > = wiki account =
> > > > The wiki where we'll be maintaining information about
> > > > boot time, and about activities of the boot time SIG, is the elinux wiki.
> > > > The page we'll be focusing on is: https://elinux.org/Boot_Time.
> > > > If you are interested in helping update and maintain the information there
> > > > (which I hope almost everyone is), then please make sure you have a user
> > > > account on the wiki.
> > > > If you don't have one, please go here:
> > > > https://elinux.org/Special:RequestAccount
> > > > I have to manually approve accounts in order to fight spambots. It might
> > > > take a few days for me to get to your request. It's very helpful if you
> > > > put a comment in one of the request fields about this being related to
> > > > the boot-time initiative or SIG, so I can distinguish your request from
> > > > spam requests.
> > >
> > > Can we instead keep this all a part of the kernel docs instead of the
> > > wiki? Couple of reasons for that:
> >
> > Ideally, we would put some material in the wiki, and also
> > produce a document - some kind of "boot-time tuning guide" that can
> > live in the kernel tree.
>
> This is the part I care most about being in the kernel docs. Eg: what
> configs to use. What commandline params to set. Dos and Don'ts for the
> drivers, etc. So, good to see that is an acceptable option.
I'm interested to help contribute to a boot speed document, and I
suspect some others at Red Hat are interested as well. Personally,
I would prefer to have a section in the kernel documentation over a
Wiki. Besides arch-specific recommendations, we can also contribute
some boot speed improvement techniques that we've done that are
specific to RT.
In addition to the recommended configs, I think it would also be
beneficial to list some upstream patches that improve boot speed along
with the kernel version it was introduced in.
Brian
^ permalink raw reply
* RE: [boot-time] RFC: proposal for boot-time tools github repository
From: Bird, Tim @ 2024-11-05 2:04 UTC (permalink / raw)
To: Francesco Valla, linux-embedded@vger.kernel.org
In-Reply-To: <2523504.XAFRqVoOGU@fedora.fritz.box>
> -----Original Message-----
> From: Francesco Valla <valla.francesco@gmail.com>
> On Wednesday, 30 October 2024 at 09:22:09 Bird, Tim <Tim.Bird@sony.com> wrote:
> > I have a few tools that I plan to publish over the next little while.
> >
> > Some of these might make their way into the kernel 'scripts' directory, but some
> > others of these might not be appropriate to add there.
> >
> > So I'm thinking of putting together a repository of boot-time related tools
> > for people to play with. Here's a sample of one tool that I find handy:
> > ----
> > #!/bin/sh
> > # sort-initcalls.sh - sort the initcalls by duration
> >
> > if [ -z "$1" -o "$1" = "-h" ] ; then
> > echo "Usage: sort-initcalls.sh <dmesg file>"
> > exit 1
> > fi
> >
> > grep "initcall.*after" $1 | sed "s/\(.*\)after\(.*\)/\2 \1/g" | sed "s/\r//" | sort -n
> > ----
> > See https://elinux.org/Initcall_Debug for details.
> >
> > For tools on their way upstream, this would serve as a development repository
> > where different ideas and techniques can get hashed out.
> >
> > So, does anyone have alternative ideas for hosting such tools, or comments on
> > this approach?
>
> Sounds fine, at least from the perspective of a developer not so much expert
> on the whole mainline development flow (like me).
>
> Were you thinking of using the Github PR method to accept contributions?
Yes, as well as patches on the mailing list in a format suitable for a kernel submission,
even though it wouldn't be going into a kernel tree, but rather a github repository
with user-space tools (and maybe a yocto layer definition).
I hope to initialize the repository this week.
-- Tim
^ permalink raw reply
* Re: [boot-time] RFC: proposal for boot-time tools github repository
From: Francesco Valla @ 2024-11-03 16:15 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org, Bird, Tim
In-Reply-To: <MW5PR13MB5632D76A85C2DCE48B5AA2B4FD542@MW5PR13MB5632.namprd13.prod.outlook.com>
On Wednesday, 30 October 2024 at 09:22:09 Bird, Tim <Tim.Bird@sony.com> wrote:
> I have a few tools that I plan to publish over the next little while.
>
> Some of these might make their way into the kernel 'scripts' directory, but some
> others of these might not be appropriate to add there.
>
> So I'm thinking of putting together a repository of boot-time related tools
> for people to play with. Here's a sample of one tool that I find handy:
> ----
> #!/bin/sh
> # sort-initcalls.sh - sort the initcalls by duration
>
> if [ -z "$1" -o "$1" = "-h" ] ; then
> echo "Usage: sort-initcalls.sh <dmesg file>"
> exit 1
> fi
>
> grep "initcall.*after" $1 | sed "s/\(.*\)after\(.*\)/\2 \1/g" | sed "s/\r//" | sort -n
> ----
> See https://elinux.org/Initcall_Debug for details.
>
> For tools on their way upstream, this would serve as a development repository
> where different ideas and techniques can get hashed out.
>
> So, does anyone have alternative ideas for hosting such tools, or comments on
> this approach?
Sounds fine, at least from the perspective of a developer not so much expert
on the whole mainline development flow (like me).
Were you thinking of using the Github PR method to accept contributions?
Greetings,
Francesco
^ permalink raw reply
* [boot-time] RFC: proposal for boot-time tools github repository
From: Bird, Tim @ 2024-10-30 8:22 UTC (permalink / raw)
To: linux-embedded@vger.kernel.org
I have a few tools that I plan to publish over the next little while.
Some of these might make their way into the kernel 'scripts' directory, but some
others of these might not be appropriate to add there.
So I'm thinking of putting together a repository of boot-time related tools
for people to play with. Here's a sample of one tool that I find handy:
----
#!/bin/sh
# sort-initcalls.sh - sort the initcalls by duration
if [ -z "$1" -o "$1" = "-h" ] ; then
echo "Usage: sort-initcalls.sh <dmesg file>"
exit 1
fi
grep "initcall.*after" $1 | sed "s/\(.*\)after\(.*\)/\2 \1/g" | sed "s/\r//" | sort -n
----
See https://elinux.org/Initcall_Debug for details.
For tools on their way upstream, this would serve as a development repository
where different ideas and techniques can get hashed out.
So, does anyone have alternative ideas for hosting such tools, or comments on
this approach?
Eventually, I expect to make a yocto recipe that would make installing
boot-time instrumentation, testing, research and tuning tools easily available.
-- Tim
^ permalink raw reply
* RE: elinux.org wiki style
From: Bird, Tim @ 2024-10-30 0:01 UTC (permalink / raw)
To: Geert Uytterhoeven; +Cc: Linux Embedded
In-Reply-To: <CAMuHMdXN3J=_xFQkuhcLxhniNK0LqTH9rFM9Ydex1TUufuBw3w@mail.gmail.com>
> -----Original Message-----
> From: Geert Uytterhoeven <geert@linux-m68k.org>
> Hi Tim,
>
> https://elinux.org/ is looking weird today.
>
> There seems to be something wrong with the stylesheet.
> When trying to open it:
>
> [b554204955cf12a33d7baee1]
> /load.php?debug=false&lang=en&modules=ext.echo.badgeicons%7Cext.echo.styles.badge%7Cmediawiki.legacy.commonPrint%2Cshar
> ed%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector
> Error from line 689 of
> /var/www/elinux.org/htdocs/includes/exception/MWExceptionHandler.php:
> Class 'FormatJson' not found
>
> Backtrace:
>
> #0 /var/www/elinux.org/htdocs/includes/exception/MWExceptionHandler.php(216):
> MWExceptionHandler::logError(ErrorException, string, string)
> #1 /var/www/elinux.org/htdocs/includes/AutoLoader.php(109):
> MWExceptionHandler::handleError(integer, string, string, integer,
> array)
> #2 /var/www/elinux.org/htdocs/includes/AutoLoader.php(109): require()
> #3 [internal function]: AutoLoader::autoload(string)
> #4 /var/www/elinux.org/htdocs/includes/resourceloader/ResourceLoader.php(141):
> spl_autoload_call(string)
> #5 /var/www/elinux.org/htdocs/includes/resourceloader/ResourceLoader.php(751):
> ResourceLoader->preloadModuleInfo(array, ResourceLoaderContext)
> #6 /var/www/elinux.org/htdocs/load.php(51):
> ResourceLoader->respond(ResourceLoaderContext)
> #7 {main}
>
Yes. I saw the format error yesterday, and asked Bill Traynor to look into it.
Thanks for the details and backtrace. I'll send those on to Bill.
-- Tim
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox