Generate the config file for kernel compilation non-interactively in script.

Generate the config file for kernel compilation non-interactively in script.

[Hongyi Zhao]

Hi,

I want to build the realtime Linux for ROS 2 according to the
guidelines here:
<https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.

For this purpose, I must enable the rt_preempt relative options in the
kernel withe the following method interactively:

$ make menuconfig

and set the following

# Enable CONFIG_PREEMPT_RT
 -> General Setup
  -> Preemption Model (Fully Preemptible Kernel (Real-Time))
   (X) Fully Preemptible Kernel (Real-Time)

# Enable CONFIG_HIGH_RES_TIMERS
 -> General setup
  -> Timers subsystem
   [*] High Resolution Timer Support

# Enable CONFIG_NO_HZ_FULL
 -> General setup
  -> Timers subsystem
   -> Timer tick handling (Full dynticks system (tickless))
    (X) Full dynticks system (tickless)

# Set CONFIG_HZ_1000 (note: this is no longer in the General Setup
menu, go back twice)
 -> Processor type and features
  -> Timer frequency (1000 HZ)
   (X) 1000 HZ

# Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
 ->  Power management and ACPI options
  -> CPU Frequency scaling
   -> CPU Frequency scaling (CPU_FREQ [=y])
    -> Default CPUFreq governor (<choice> [=y])
     (X) performance

But this is very inconvenient for doing the above job in script. Is
there an alternative method to generate the above configurations for
kernel compilation  non-interactively in script.

BR,
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Generate the config file for kernel compilation non-interactively in script.

[Randy Dunlap]

On 12/31/20 8:51 PM, Hongyi Zhao wrote:
> Hi,
> 
> I want to build the realtime Linux for ROS 2 according to the
> guidelines here:
> <https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.
> 
> For this purpose, I must enable the rt_preempt relative options in the
> kernel withe the following method interactively:
> 
> $ make menuconfig
> 
> and set the following
> 
> # Enable CONFIG_PREEMPT_RT
>  -> General Setup
>   -> Preemption Model (Fully Preemptible Kernel (Real-Time))
>    (X) Fully Preemptible Kernel (Real-Time)
> 
> # Enable CONFIG_HIGH_RES_TIMERS
>  -> General setup
>   -> Timers subsystem
>    [*] High Resolution Timer Support
> 
> # Enable CONFIG_NO_HZ_FULL
>  -> General setup
>   -> Timers subsystem
>    -> Timer tick handling (Full dynticks system (tickless))
>     (X) Full dynticks system (tickless)
> 
> # Set CONFIG_HZ_1000 (note: this is no longer in the General Setup
> menu, go back twice)
>  -> Processor type and features
>   -> Timer frequency (1000 HZ)
>    (X) 1000 HZ
> 
> # Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
>  ->  Power management and ACPI options
>   -> CPU Frequency scaling
>    -> CPU Frequency scaling (CPU_FREQ [=y])
>     -> Default CPUFreq governor (<choice> [=y])
>      (X) performance
> 
> But this is very inconvenient for doing the above job in script. Is
> there an alternative method to generate the above configurations for
> kernel compilation  non-interactively in script.

Hi,
You can use scripts/config in the kernel source tree.
Something like this (I don't have RT kernel sources):


scripts/config -e PREEMPT_RT
scripts/config -e HIGH_RES_TIMERS
scripts/config -e NO_HZ_FULL
scripts/config -e HZ_1000
scripts/config -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE


Note that if any of those have other Kconfig dependencies, those Kconfig
symbols will also have to be enabled for this to work.

And then run 'make oldconfig' to update the kernel .config file.


HTH.
-- 
~Randy

Re: Generate the config file for kernel compilation non-interactively in script.

[Hongyi Zhao]

On Fri, Jan 1, 2021 at 2:32 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>
> On 12/31/20 8:51 PM, Hongyi Zhao wrote:
> > Hi,
> >
> > I want to build the realtime Linux for ROS 2 according to the
> > guidelines here:
> > <https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.
> >
> > For this purpose, I must enable the rt_preempt relative options in the
> > kernel withe the following method interactively:
> >
> > $ make menuconfig
> >
> > and set the following
> >
> > # Enable CONFIG_PREEMPT_RT
> >  -> General Setup
> >   -> Preemption Model (Fully Preemptible Kernel (Real-Time))
> >    (X) Fully Preemptible Kernel (Real-Time)
> >
> > # Enable CONFIG_HIGH_RES_TIMERS
> >  -> General setup
> >   -> Timers subsystem
> >    [*] High Resolution Timer Support
> >
> > # Enable CONFIG_NO_HZ_FULL
> >  -> General setup
> >   -> Timers subsystem
> >    -> Timer tick handling (Full dynticks system (tickless))
> >     (X) Full dynticks system (tickless)
> >
> > # Set CONFIG_HZ_1000 (note: this is no longer in the General Setup
> > menu, go back twice)
> >  -> Processor type and features
> >   -> Timer frequency (1000 HZ)
> >    (X) 1000 HZ
> >
> > # Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
> >  ->  Power management and ACPI options
> >   -> CPU Frequency scaling
> >    -> CPU Frequency scaling (CPU_FREQ [=y])
> >     -> Default CPUFreq governor (<choice> [=y])
> >      (X) performance
> >
> > But this is very inconvenient for doing the above job in script. Is
> > there an alternative method to generate the above configurations for
> > kernel compilation  non-interactively in script.
>
> Hi,
> You can use scripts/config in the kernel source tree.
> Something like this (I don't have RT kernel sources):
>
>
> scripts/config -e PREEMPT_RT
> scripts/config -e HIGH_RES_TIMERS
> scripts/config -e NO_HZ_FULL
> scripts/config -e HZ_1000
> scripts/config -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE

Wonderful. Thanks a lot for your instructions. I really have noticed
this tool but failed to figure out the corresponding translation rules
for the options used by menuconfig and this script.

BTW, how do you figure out the above options/arguments corresponding
to the ones I've mentioned previously?

>
> Note that if any of those have other Kconfig dependencies, those Kconfig
> symbols will also have to be enabled for this to work.

How to know whether an option has other Kconfig dependencies and find
the corresponding symbols/arguments for feeding to scripts/config?

> And then run 'make oldconfig' to update the kernel .config file.

Thanks again.

BR,
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Generate the config file for kernel compilation non-interactively in script.

[Randy Dunlap]

Hi,

On 1/1/21 1:55 AM, Hongyi Zhao wrote:
> On Fri, Jan 1, 2021 at 2:32 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>>
>> On 12/31/20 8:51 PM, Hongyi Zhao wrote:
>>> Hi,
>>>
>>> I want to build the realtime Linux for ROS 2 according to the
>>> guidelines here:
>>> <https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.
>>>
>>> For this purpose, I must enable the rt_preempt relative options in the
>>> kernel withe the following method interactively:
>>>
>>> $ make menuconfig
>>>
>>> and set the following
>>>
>>> # Enable CONFIG_PREEMPT_RT
>>>  -> General Setup
>>>   -> Preemption Model (Fully Preemptible Kernel (Real-Time))
>>>    (X) Fully Preemptible Kernel (Real-Time)
>>>
>>> # Enable CONFIG_HIGH_RES_TIMERS
>>>  -> General setup
>>>   -> Timers subsystem
>>>    [*] High Resolution Timer Support
>>>
>>> # Enable CONFIG_NO_HZ_FULL
>>>  -> General setup
>>>   -> Timers subsystem
>>>    -> Timer tick handling (Full dynticks system (tickless))
>>>     (X) Full dynticks system (tickless)
>>>
>>> # Set CONFIG_HZ_1000 (note: this is no longer in the General Setup
>>> menu, go back twice)
>>>  -> Processor type and features
>>>   -> Timer frequency (1000 HZ)
>>>    (X) 1000 HZ
>>>
>>> # Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
>>>  ->  Power management and ACPI options
>>>   -> CPU Frequency scaling
>>>    -> CPU Frequency scaling (CPU_FREQ [=y])
>>>     -> Default CPUFreq governor (<choice> [=y])
>>>      (X) performance
>>>
>>> But this is very inconvenient for doing the above job in script. Is
>>> there an alternative method to generate the above configurations for
>>> kernel compilation  non-interactively in script.
>>
>> Hi,
>> You can use scripts/config in the kernel source tree.
>> Something like this (I don't have RT kernel sources):
>>
>>
>> scripts/config -e PREEMPT_RT
>> scripts/config -e HIGH_RES_TIMERS
>> scripts/config -e NO_HZ_FULL
>> scripts/config -e HZ_1000
>> scripts/config -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE
> 
> Wonderful. Thanks a lot for your instructions. I really have noticed
> this tool but failed to figure out the corresponding translation rules
> for the options used by menuconfig and this script.
> 
> BTW, how do you figure out the above options/arguments corresponding
> to the ones I've mentioned previously?
> 

Oh, I just took the ones that you had listed and removed the leading
"CONFIG_" from them.

>>
>> Note that if any of those have other Kconfig dependencies, those Kconfig
>> symbols will also have to be enabled for this to work.
> 
> How to know whether an option has other Kconfig dependencies and find
> the corresponding symbols/arguments for feeding to scripts/config?

Use one of the interactive config tools (nconfig, xconfig).
They will show you dependencies, but you may have to enable other
symbols first.

Maybe it would be easier to do a temporary 'make allmodconfig'
to have the symbols that you are interested in be enabled, then
you can find them and look at their dependencies.

Or you could just read the Kconfig files, but that would probably
be painful.

>> And then run 'make oldconfig' to update the kernel .config file.
> 
> Thanks again.


-- 
~Randy

Re: Generate the config file for kernel compilation non-interactively in script.

[Hongyi Zhao]

On Sat, Jan 2, 2021 at 12:28 AM Randy Dunlap <rdunlap@infradead.org> wrote:
>
> Hi,
>
> On 1/1/21 1:55 AM, Hongyi Zhao wrote:
> > On Fri, Jan 1, 2021 at 2:32 PM Randy Dunlap <rdunlap@infradead.org> wrote:
> >>
> >> On 12/31/20 8:51 PM, Hongyi Zhao wrote:
> >>> Hi,
> >>>
> >>> I want to build the realtime Linux for ROS 2 according to the
> >>> guidelines here:
> >>> <https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.
> >>>
> >>> For this purpose, I must enable the rt_preempt relative options in the
> >>> kernel withe the following method interactively:
> >>>
> >>> $ make menuconfig
> >>>
> >>> and set the following
> >>>
> >>> # Enable CONFIG_PREEMPT_RT
> >>>  -> General Setup
> >>>   -> Preemption Model (Fully Preemptible Kernel (Real-Time))
> >>>    (X) Fully Preemptible Kernel (Real-Time)
> >>>
> >>> # Enable CONFIG_HIGH_RES_TIMERS
> >>>  -> General setup
> >>>   -> Timers subsystem
> >>>    [*] High Resolution Timer Support
> >>>
> >>> # Enable CONFIG_NO_HZ_FULL
> >>>  -> General setup
> >>>   -> Timers subsystem
> >>>    -> Timer tick handling (Full dynticks system (tickless))
> >>>     (X) Full dynticks system (tickless)
> >>>
> >>> # Set CONFIG_HZ_1000 (note: this is no longer in the General Setup
> >>> menu, go back twice)
> >>>  -> Processor type and features
> >>>   -> Timer frequency (1000 HZ)
> >>>    (X) 1000 HZ
> >>>
> >>> # Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
> >>>  ->  Power management and ACPI options
> >>>   -> CPU Frequency scaling
> >>>    -> CPU Frequency scaling (CPU_FREQ [=y])
> >>>     -> Default CPUFreq governor (<choice> [=y])
> >>>      (X) performance
> >>>
> >>> But this is very inconvenient for doing the above job in script. Is
> >>> there an alternative method to generate the above configurations for
> >>> kernel compilation  non-interactively in script.
> >>
> >> Hi,
> >> You can use scripts/config in the kernel source tree.
> >> Something like this (I don't have RT kernel sources):
> >>
> >>
> >> scripts/config -e PREEMPT_RT
> >> scripts/config -e HIGH_RES_TIMERS
> >> scripts/config -e NO_HZ_FULL
> >> scripts/config -e HZ_1000
> >> scripts/config -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE
> >
> > Wonderful. Thanks a lot for your instructions. I really have noticed
> > this tool but failed to figure out the corresponding translation rules
> > for the options used by menuconfig and this script.
> >
> > BTW, how do you figure out the above options/arguments corresponding
> > to the ones I've mentioned previously?
> >
>
> Oh, I just took the ones that you had listed and removed the leading
> "CONFIG_" from them.
>
> >>
> >> Note that if any of those have other Kconfig dependencies, those Kconfig
> >> symbols will also have to be enabled for this to work.
> >
> > How to know whether an option has other Kconfig dependencies and find
> > the corresponding symbols/arguments for feeding to scripts/config?
>
> Use one of the interactive config tools (nconfig, xconfig).
> They will show you dependencies, but you may have to enable other
> symbols first.
>
> Maybe it would be easier to do a temporary 'make allmodconfig'
> to have the symbols that you are interested in be enabled, then
> you can find them and look at their dependencies.

It sounds still complicated for manually operation of the above
mentioned procedure even by the virtue of scripts/config. The more
feasible way should be done with python package/binding/library
programmatically, but I'm not sure whether such stuff exists.

BR,
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Generate the config file for kernel compilation non-interactively in script.

[Hongyi Zhao]

On Fri, Jan 1, 2021 at 2:32 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>
> On 12/31/20 8:51 PM, Hongyi Zhao wrote:
> > Hi,
> >
> > I want to build the realtime Linux for ROS 2 according to the
> > guidelines here:
> > <https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.
> >
> > For this purpose, I must enable the rt_preempt relative options in the
> > kernel withe the following method interactively:
> >
> > $ make menuconfig
> >
> > and set the following
> >
> > # Enable CONFIG_PREEMPT_RT
> >  -> General Setup
> >   -> Preemption Model (Fully Preemptible Kernel (Real-Time))
> >    (X) Fully Preemptible Kernel (Real-Time)
> >
> > # Enable CONFIG_HIGH_RES_TIMERS
> >  -> General setup
> >   -> Timers subsystem
> >    [*] High Resolution Timer Support
> >
> > # Enable CONFIG_NO_HZ_FULL
> >  -> General setup
> >   -> Timers subsystem
> >    -> Timer tick handling (Full dynticks system (tickless))
> >     (X) Full dynticks system (tickless)
> >
> > # Set CONFIG_HZ_1000 (note: this is no longer in the General Setup
> > menu, go back twice)
> >  -> Processor type and features
> >   -> Timer frequency (1000 HZ)
> >    (X) 1000 HZ
> >
> > # Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
> >  ->  Power management and ACPI options
> >   -> CPU Frequency scaling
> >    -> CPU Frequency scaling (CPU_FREQ [=y])
> >     -> Default CPUFreq governor (<choice> [=y])
> >      (X) performance
> >
> > But this is very inconvenient for doing the above job in script. Is
> > there an alternative method to generate the above configurations for
> > kernel compilation  non-interactively in script.
>
> Hi,
> You can use scripts/config in the kernel source tree.
> Something like this (I don't have RT kernel sources):
>
>
> scripts/config -e PREEMPT_RT
> scripts/config -e HIGH_RES_TIMERS
> scripts/config -e NO_HZ_FULL
> scripts/config -e HZ_1000
> scripts/config -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE

Based on the built-in help of this script:

$ bash scripts/config -h |& egrep -- '-e |--file|commands'
commands:
    --enable|-e option   Enable option
    commands can be repeated multiple times
    --file config-file   .config file to change (default .config)

It's obviously that the above commands you suggested can be combined
into one as shown below:

$ scripts/config --file .config  -e PREEMPT_RT -e HIGH_RES_TIMERS -e
NO_HZ_FULL -e HZ_1000 -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE

or simply:

$ scripts/config -e PREEMPT_RT -e HIGH_RES_TIMERS -e NO_HZ_FULL -e
HZ_1000 -e CPU_FREQ_DEFAULT_GOV_PERFORMANCE

BR,

>
>
> Note that if any of those have other Kconfig dependencies, those Kconfig
> symbols will also have to be enabled for this to work.
>
> And then run 'make oldconfig' to update the kernel .config file.
>
>
> HTH.
> --
> ~Randy
>


-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: Generate the config file for kernel compilation non-interactively in script.

[Theodore Ts'o]

On Fri, Jan 01, 2021 at 12:51:13PM +0800, Hongyi Zhao wrote:
> 
> I want to build the realtime Linux for ROS 2 according to the
> guidelines here:
> <https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2/>.
> 
> For this purpose, I must enable the rt_preempt relative options in the
> kernel withe the following method interactively:
> 
> But this is very inconvenient for doing the above job in script. Is
> there an alternative method to generate the above configurations for
> kernel compilation  non-interactively in script.

What I do for a slightly different use case is to use defconfigs.
After setting up a minimum kernel that has what I need for my use case
(in my case, to build a kernel that works for kvm-xfstests and
gce-xfststs), I save a defconfig: "make savedefconfig", and then stash
in a git repository:

https://github.com/tytso/xfstests-bld/tree/master/kernel-configs

Then when I need to build a kernel, I just copy a defconfig to
.config, and then run the command "make olddefconfig" to expand it
out.  The reason why I use defconfigs is that very often, at least for
my use cases, a defconfig generated for kernel version X.Y tends to
work for X.Y+1, X.Y+2, etc.  That's not always true, of course, which
is why there are a few extra lines added to:

https://github.com/tytso/xfstests-bld/blob/master/kernel-configs/x86_64-config-4.19

This was needed so that the this defconfig will work for 4.19.y
through 5.3.y.  The special cases were needed for 5.1 and 5.2, but I
haven't needed it for any other kernel versions in terms of making a
kernel that would correctly boot on KVM and GCE and correctly run
xfstests for ext4, xfs, btrfs, f2fs, etc.  So I just create a
defconfig for each LTS kernel version, and for the most part, it will
work for future kernel versions until the next LTS kerenl version gets
released, at which point I create a new defconfig for my test
appliance use case.

If your goal is to build newer kernel versions for RT_PREMPT kernels
for your use case, perhaps this technique will be helpful for you.

Hope this helps,

					- Ted