[MODALTS v0.1 4/4] add modules deps alternatives description

[MODALTS v0.1 4/4] add modules deps alternatives description

[Valerii Chernous]

Cc: xe-linux-external@cisco.com
Cc: Valerii Chernous <vchernou@cisco.com>
Signed-off-by: Valerii Chernous <vchernou@cisco.com>
---
 README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
 create mode 100644 README.deps.alternatives.txt

diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
new file mode 100644
index 0000000..9ad3ce5
--- /dev/null
+++ b/README.deps.alternatives.txt
@@ -0,0 +1,40 @@
+Modules alternatives feature allow to calculate dependency alternatives
+during build time and aviod to regenerate modules db into runtime
+
+To enable deps alternatives calculation use "-D" flag with depmod,
+it will create indexes modules.alternatives and modules.alternatives.bin
+This indexes will be used by modprobe in runtime
+By default modprobe will load first(primary/major) dependency from list
+If it required to load alternative module, it should be done manually before
+loading main modules set.
+For example systemd service that detect platform type can load required platform
+modules and after it run main device initialization
+In case when alternative module loaded, modprobe detect this and skip to load primary
+dependency
+
+modules deps alternatives generation basic algorithm description
+1. Load modules information(imported/exported symbols)
+2. Find depended symbol alternatives(create list available symbols
+   alternatives instead of storing last one)
+3. Choise primary/major alternative per depended symbol correspond to
+   build time dependency(build time deps getting from module info section)
+4. Create a list of dependency modules alternatives correspond to next rule:
+4.1 All modules alternatives for module dependency should provide all symbols
+    from primary/major dependency module
+5 Store modules alternatives index(modules.alternatives) as key:value where
+key is a pair depended#_#primary_depency,
+value is list of all modules that provide all symbols from primary_depency
+for depended module
+
+Note:
+Current implementation/algorithm doesn't cover variant where requred symbols
+from primary deps provided by more that one modules. Exporting all symbols in
+alternative depency that used by depended module from primary_depency is
+mandatory!
+
+Note:
+modules.dep index different for standard/basic and modules alternatives algorithms
+modules.dep for modules alternatives algorithm contain only direct dependencies and
+full dependency list will be calculated into runtime correspond to preferred alternative.
+modules.dep for standard/basic algorithm contain full dependency list for module and
+can't be changed during runtime without rebuild database via depmod
-- 
2.35.6

Re: [MODALTS v0.1 4/4] add modules deps alternatives description

[Lucas De Marchi]

On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
>Cc: xe-linux-external@cisco.com
>Cc: Valerii Chernous <vchernou@cisco.com>
>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
>---
> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 40 insertions(+)
> create mode 100644 README.deps.alternatives.txt
>
>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
>new file mode 100644
>index 0000000..9ad3ce5
>--- /dev/null
>+++ b/README.deps.alternatives.txt
>@@ -0,0 +1,40 @@
>+Modules alternatives feature allow to calculate dependency alternatives
>+during build time and aviod to regenerate modules db into runtime
>+
>+To enable deps alternatives calculation use "-D" flag with depmod,
>+it will create indexes modules.alternatives and modules.alternatives.bin
>+This indexes will be used by modprobe in runtime
>+By default modprobe will load first(primary/major) dependency from list
>+If it required to load alternative module, it should be done manually before
>+loading main modules set.
>+For example systemd service that detect platform type can load required platform
>+modules and after it run main device initialization
>+In case when alternative module loaded, modprobe detect this and skip to load primary
>+dependency
>+
>+modules deps alternatives generation basic algorithm description
>+1. Load modules information(imported/exported symbols)
>+2. Find depended symbol alternatives(create list available symbols
>+   alternatives instead of storing last one)
>+3. Choise primary/major alternative per depended symbol correspond to
>+   build time dependency(build time deps getting from module info section)
>+4. Create a list of dependency modules alternatives correspond to next rule:
>+4.1 All modules alternatives for module dependency should provide all symbols
>+    from primary/major dependency module
>+5 Store modules alternatives index(modules.alternatives) as key:value where
>+key is a pair depended#_#primary_depency,
>+value is list of all modules that provide all symbols from primary_depency
>+for depended module
>+
>+Note:
>+Current implementation/algorithm doesn't cover variant where requred symbols
>+from primary deps provided by more that one modules. Exporting all symbols in
>+alternative depency that used by depended module from primary_depency is
>+mandatory!
>+
>+Note:
>+modules.dep index different for standard/basic and modules alternatives algorithms
>+modules.dep for modules alternatives algorithm contain only direct dependencies and
>+full dependency list will be calculated into runtime correspond to preferred alternative.
>+modules.dep for standard/basic algorithm contain full dependency list for module and
>+can't be changed during runtime without rebuild database via depmod


well... this kind of explains the what, but still no clue on why.
If multiple different modules are providing the same symbol, then they
are doing things wrong.

If there are multiple variants of the same module (again, is this about
external modules?), then I see no advantage to delay the decisions from
depmod-time to modprobe-time. Just setup your depmod.d configuration.

Also end users have not visibility on a README.deps.alternatives.txt
file. Documentation in kmod is kept on man pages.


Lucas De Marchi

>-- 
>2.35.6
>

Re: [MODALTS v0.1 4/4] add modules deps alternatives description

[Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)]

>On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
>>Cc: xe-linux-external@cisco.com
>>Cc: Valerii Chernous <vchernou@cisco.com>
>>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
>>---
>> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
>> 1 file changed, 40 insertions(+)
>> create mode 100644 README.deps.alternatives.txt
>>
>>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
>>new file mode 100644
>>index 0000000..9ad3ce5
>>--- /dev/null
>>+++ b/README.deps.alternatives.txt
>>@@ -0,0 +1,40 @@
>>+Modules alternatives feature allow to calculate dependency alternatives
>>+during build time and aviod to regenerate modules db into runtime
>>+
>>+To enable deps alternatives calculation use "-D" flag with depmod,
>>+it will create indexes modules.alternatives and modules.alternatives.bin
>>+This indexes will be used by modprobe in runtime
>>+By default modprobe will load first(primary/major) dependency from list
>>+If it required to load alternative module, it should be done manually before
>>+loading main modules set.
>>+For example systemd service that detect platform type can load required platform
>>+modules and after it run main device initialization
>>+In case when alternative module loaded, modprobe detect this and skip to load primary
>>+dependency
>>+
>>+modules deps alternatives generation basic algorithm description
>>+1. Load modules information(imported/exported symbols)
>>+2. Find depended symbol alternatives(create list available symbols
>>+   alternatives instead of storing last one)
>>+3. Choise primary/major alternative per depended symbol correspond to
>>+   build time dependency(build time deps getting from module info section)
>>+4. Create a list of dependency modules alternatives correspond to next rule:
>>+4.1 All modules alternatives for module dependency should provide all symbols
>>+5 Store modules alternatives index(modules.alternatives) as key:value where
>>+key is a pair depended#_#primary_depency,
>>+value is list of all modules that provide all symbols from primary_depency
>>+for depended module
>>+
>>+Note:
>>+Current implementation/algorithm doesn't cover variant where requred symbols
>>+from primary deps provided by more that one modules. Exporting all symbols in
>>+alternative depency that used by depended module from primary_depency is
>>+mandatory!
>>+
>>+Note:
>>+modules.dep index different for standard/basic and modules alternatives algorithms
>>+modules.dep for modules alternatives algorithm contain only direct dependencies and
>>+full dependency list will be calculated into runtime correspond to preferred alternative.
>>+modules.dep for standard/basic algorithm contain full dependency list for module and
>>+can't be changed during runtime without rebuild database via depmod


>well... this kind of explains the what, but still no clue on why.
>If multiple different modules are providing the same symbol, then they
>are doing things wrong.

>If there are multiple variants of the same module (again, is this about
>external modules?), then I see no advantage to delay the decisions from
>depmod-time to modprobe-time. Just setup your depmod.d configuration.

>Also end users have not visibility on a README.deps.alternatives.txt
>file. Documentation in kmod is kept on man pages.


>Lucas De Marchi

First at all, thank you for review, Lucas.
Let me try to explain feature more:
1. You are correct, feature tested on external modules

2.
>If multiple different modules are providing the same symbol, then they
>are doing things wrong.

Modules exported the same api(the same functions) and on my opinion it's ok
and kernel process normally different modules with the same exports. One major
restriction is only one module with the same symbols can be loaded on the same
time but it's ok in my case(as I described, in my case, it's per platform
modules and devices with different hardware using the same software image).

3.
>If there are multiple variants of the same module (again, is this about
>external modules?), then I see no advantage to delay the decisions from
>depmod-time to modprobe-time. Just setup your depmod.d configuration.

It can be different variant of the same module but maybe not. For example it
can be cryptography modules. Modules provide the same api but implementation
of api is totally different and depend on specific hardware. With modules
alternatives feature it's easy to use this kind of modules. You can use
required alternative for specific hardware and all depended modules can use
external functions directly without any wrappers or "if" statements to resolve
dependencies.
With using depmod.d configuration it's possible to choose primary alternative
into build time but in my case required alternative is unknown during build time,
it will be known only into runtime. In this case it required to regenerate
modules db into runtime and I try to avoid this.

4.
>Also end users have not visibility on a README.deps.alternatives.txt
>file. Documentation in kmod is kept on man pages.

I can rework feature description to man pages format but at this moment for me
important to know is modules alternatives feature acceptable for kmod project?
If yes, and finally it will be merged I'm ready to spend more time to
working on it

Best regards,
Valerii

________________________________________
From: Lucas De Marchi <lucas.demarchi@intel.com>
Sent: Friday, June 28, 2024 11:55 PM
To: Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)
Cc: linux-modules@vger.kernel.org; xe-linux-external(mailer list); Nicolas Schier; Lucas De Marchi
Subject: Re: [MODALTS v0.1 4/4] add modules deps alternatives description

On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
>Cc: xe-linux-external@cisco.com
>Cc: Valerii Chernous <vchernou@cisco.com>
>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
>---
> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 40 insertions(+)
> create mode 100644 README.deps.alternatives.txt
>
>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
>new file mode 100644
>index 0000000..9ad3ce5
>--- /dev/null
>+++ b/README.deps.alternatives.txt
>@@ -0,0 +1,40 @@
>+Modules alternatives feature allow to calculate dependency alternatives
>+during build time and aviod to regenerate modules db into runtime
>+
>+To enable deps alternatives calculation use "-D" flag with depmod,
>+it will create indexes modules.alternatives and modules.alternatives.bin
>+This indexes will be used by modprobe in runtime
>+By default modprobe will load first(primary/major) dependency from list
>+If it required to load alternative module, it should be done manually before
>+loading main modules set.
>+For example systemd service that detect platform type can load required platform
>+modules and after it run main device initialization
>+In case when alternative module loaded, modprobe detect this and skip to load primary
>+dependency
>+
>+modules deps alternatives generation basic algorithm description
>+1. Load modules information(imported/exported symbols)
>+2. Find depended symbol alternatives(create list available symbols
>+   alternatives instead of storing last one)
>+3. Choise primary/major alternative per depended symbol correspond to
>+   build time dependency(build time deps getting from module info section)
>+4. Create a list of dependency modules alternatives correspond to next rule:
>+4.1 All modules alternatives for module dependency should provide all symbols
>+    from primary/major dependency module
>+5 Store modules alternatives index(modules.alternatives) as key:value where
>+key is a pair depended#_#primary_depency,
>+value is list of all modules that provide all symbols from primary_depency
>+for depended module
>+
>+Note:
>+Current implementation/algorithm doesn't cover variant where requred symbols
>+from primary deps provided by more that one modules. Exporting all symbols in
>+alternative depency that used by depended module from primary_depency is
>+mandatory!
>+
>+Note:
>+modules.dep index different for standard/basic and modules alternatives algorithms
>+modules.dep for modules alternatives algorithm contain only direct dependencies and
>+full dependency list will be calculated into runtime correspond to preferred alternative.
>+modules.dep for standard/basic algorithm contain full dependency list for module and
>+can't be changed during runtime without rebuild database via depmod


well... this kind of explains the what, but still no clue on why.
If multiple different modules are providing the same symbol, then they
are doing things wrong.

If there are multiple variants of the same module (again, is this about
external modules?), then I see no advantage to delay the decisions from
depmod-time to modprobe-time. Just setup your depmod.d configuration.

Also end users have not visibility on a README.deps.alternatives.txt
file. Documentation in kmod is kept on man pages.


Lucas De Marchi

>--
>2.35.6
>

Re: [MODALTS v0.1 4/4] add modules deps alternatives description

[Nicolas Schier]

On Mon, Jul 01, 2024 at 09:23:03AM +0000, Valerii Chernous wrote:
> >On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
> >>Cc: xe-linux-external@cisco.com
> >>Cc: Valerii Chernous <vchernou@cisco.com>
> >>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
> >>---
> >> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 40 insertions(+)
> >> create mode 100644 README.deps.alternatives.txt
> >>
> >>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
> >>new file mode 100644
> >>index 0000000..9ad3ce5
> >>--- /dev/null
> >>+++ b/README.deps.alternatives.txt
> >>@@ -0,0 +1,40 @@
> >>+Modules alternatives feature allow to calculate dependency alternatives
> >>+during build time and aviod to regenerate modules db into runtime
> >>+
> >>+To enable deps alternatives calculation use "-D" flag with depmod,
> >>+it will create indexes modules.alternatives and modules.alternatives.bin
> >>+This indexes will be used by modprobe in runtime
> >>+By default modprobe will load first(primary/major) dependency from list
> >>+If it required to load alternative module, it should be done manually before
> >>+loading main modules set.
> >>+For example systemd service that detect platform type can load required platform
> >>+modules and after it run main device initialization
> >>+In case when alternative module loaded, modprobe detect this and skip to load primary
> >>+dependency
> >>+
> >>+modules deps alternatives generation basic algorithm description
> >>+1. Load modules information(imported/exported symbols)
> >>+2. Find depended symbol alternatives(create list available symbols
> >>+   alternatives instead of storing last one)
> >>+3. Choise primary/major alternative per depended symbol correspond to
> >>+   build time dependency(build time deps getting from module info section)
> >>+4. Create a list of dependency modules alternatives correspond to next rule:
> >>+4.1 All modules alternatives for module dependency should provide all symbols
> >>+5 Store modules alternatives index(modules.alternatives) as key:value where
> >>+key is a pair depended#_#primary_depency,
> >>+value is list of all modules that provide all symbols from primary_depency
> >>+for depended module
> >>+
> >>+Note:
> >>+Current implementation/algorithm doesn't cover variant where requred symbols
> >>+from primary deps provided by more that one modules. Exporting all symbols in
> >>+alternative depency that used by depended module from primary_depency is
> >>+mandatory!
> >>+
> >>+Note:
> >>+modules.dep index different for standard/basic and modules alternatives algorithms
> >>+modules.dep for modules alternatives algorithm contain only direct dependencies and
> >>+full dependency list will be calculated into runtime correspond to preferred alternative.
> >>+modules.dep for standard/basic algorithm contain full dependency list for module and
> >>+can't be changed during runtime without rebuild database via depmod
> 
> 
> >well... this kind of explains the what, but still no clue on why.
> >If multiple different modules are providing the same symbol, then they
> >are doing things wrong.
> 
> >If there are multiple variants of the same module (again, is this about
> >external modules?), then I see no advantage to delay the decisions from
> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
> 
> >Also end users have not visibility on a README.deps.alternatives.txt
> >file. Documentation in kmod is kept on man pages.
> 
> 
> >Lucas De Marchi
> 
> First at all, thank you for review, Lucas.
> Let me try to explain feature more:
> 1. You are correct, feature tested on external modules
> 
> 2.
> >If multiple different modules are providing the same symbol, then they
> >are doing things wrong.
> 
> Modules exported the same api(the same functions) and on my opinion it's ok
> and kernel process normally different modules with the same exports. One major
> restriction is only one module with the same symbols can be loaded on the same
> time but it's ok in my case(as I described, in my case, it's per platform
> modules and devices with different hardware using the same software image).
> 
> 3.
> >If there are multiple variants of the same module (again, is this about
> >external modules?), then I see no advantage to delay the decisions from
> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
> 
> It can be different variant of the same module but maybe not. For example it
> can be cryptography modules. Modules provide the same api but implementation
> of api is totally different and depend on specific hardware. With modules
> alternatives feature it's easy to use this kind of modules. You can use
> required alternative for specific hardware and all depended modules can use
> external functions directly without any wrappers or "if" statements to resolve
> dependencies.
> With using depmod.d configuration it's possible to choose primary alternative
> into build time but in my case required alternative is unknown during build time,
> it will be known only into runtime. In this case it required to regenerate
> modules db into runtime and I try to avoid this.

It sounds to me, as if you would like to auto-generate modprobe.d/ files
for your platforms at boot time and implement the pre-loading of some
modules before some others with common modprobe.d syntax (e.g.
'install', cp. modprobe.d(5)).  But you probably evaluated that before
implementing your patches?

If your module load order is just platform dependent, I do not yet
understand, why (possibly boot-time dynamic) depmod.d configuration is
not sufficiently flexible enough.  I probably have missing some
important point.

I strongly support Lucas' request for a cover-letter.

Kind regards,
Nicolas

Re: [MODALTS v0.1 4/4] add modules deps alternatives description

[Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)]

Hi Nicolas,

> It sounds to me, as if you would like to auto-generate modprobe.d/ files
> for your platforms at boot time and implement the pre-loading of some
> modules before some others with common modprobe.d syntax (e.g.
> 'install', cp. modprobe.d(5)).  But you probably evaluated that before
> implementing your patches?

Some kind but not exactly. I try to provide method to avoid regenerating
modules deps db during runtime

> If your module load order is just platform dependent, I do not yet
> understand, why (possibly boot-time dynamic) depmod.d configuration is
> not sufficiently flexible enough.  I probably have missing some
> important point.

Device filesystem mounted as readonly to reduce security riscs so using
depmod during boot time is restricted

> I strongly support Lucas' request for a cover-letter.
Probably, it will be a good idea to prepare test repo with couple of simple
modules to demonstrate what patchset do

> Kind regards,
> Nicolas

Thank you Nicolas for review :)
Best regards,
Valerii

________________________________________
From: Nicolas Schier <n.schier@avm.de>
Sent: Monday, July 1, 2024 1:33 PM
To: Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)
Cc: Lucas De Marchi; linux-modules@vger.kernel.org; xe-linux-external(mailer list); Lucas De Marchi
Subject: Re: [MODALTS v0.1 4/4] add modules deps alternatives description

On Mon, Jul 01, 2024 at 09:23:03AM +0000, Valerii Chernous wrote:
> >On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
> >>Cc: xe-linux-external@cisco.com
> >>Cc: Valerii Chernous <vchernou@cisco.com>
> >>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
> >>---
> >> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 40 insertions(+)
> >> create mode 100644 README.deps.alternatives.txt
> >>
> >>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
> >>new file mode 100644
> >>index 0000000..9ad3ce5
> >>--- /dev/null
> >>+++ b/README.deps.alternatives.txt
> >>@@ -0,0 +1,40 @@
> >>+Modules alternatives feature allow to calculate dependency alternatives
> >>+during build time and aviod to regenerate modules db into runtime
> >>+
> >>+To enable deps alternatives calculation use "-D" flag with depmod,
> >>+it will create indexes modules.alternatives and modules.alternatives.bin
> >>+This indexes will be used by modprobe in runtime
> >>+By default modprobe will load first(primary/major) dependency from list
> >>+If it required to load alternative module, it should be done manually before
> >>+loading main modules set.
> >>+For example systemd service that detect platform type can load required platform
> >>+modules and after it run main device initialization
> >>+In case when alternative module loaded, modprobe detect this and skip to load primary
> >>+dependency
> >>+
> >>+modules deps alternatives generation basic algorithm description
> >>+1. Load modules information(imported/exported symbols)
> >>+2. Find depended symbol alternatives(create list available symbols
> >>+   alternatives instead of storing last one)
> >>+3. Choise primary/major alternative per depended symbol correspond to
> >>+   build time dependency(build time deps getting from module info section)
> >>+4. Create a list of dependency modules alternatives correspond to next rule:
> >>+4.1 All modules alternatives for module dependency should provide all symbols
> >>+5 Store modules alternatives index(modules.alternatives) as key:value where
> >>+key is a pair depended#_#primary_depency,
> >>+value is list of all modules that provide all symbols from primary_depency
> >>+for depended module
> >>+
> >>+Note:
> >>+Current implementation/algorithm doesn't cover variant where requred symbols
> >>+from primary deps provided by more that one modules. Exporting all symbols in
> >>+alternative depency that used by depended module from primary_depency is
> >>+mandatory!
> >>+
> >>+Note:
> >>+modules.dep index different for standard/basic and modules alternatives algorithms
> >>+modules.dep for modules alternatives algorithm contain only direct dependencies and
> >>+full dependency list will be calculated into runtime correspond to preferred alternative.
> >>+modules.dep for standard/basic algorithm contain full dependency list for module and
> >>+can't be changed during runtime without rebuild database via depmod
>
>
> >well... this kind of explains the what, but still no clue on why.
> >If multiple different modules are providing the same symbol, then they
> >are doing things wrong.
>
> >If there are multiple variants of the same module (again, is this about
> >external modules?), then I see no advantage to delay the decisions from
> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
>
> >Also end users have not visibility on a README.deps.alternatives.txt
> >file. Documentation in kmod is kept on man pages.
>
>
> >Lucas De Marchi
>
> First at all, thank you for review, Lucas.
> Let me try to explain feature more:
> 1. You are correct, feature tested on external modules
>
> 2.
> >If multiple different modules are providing the same symbol, then they
> >are doing things wrong.
>
> Modules exported the same api(the same functions) and on my opinion it's ok
> and kernel process normally different modules with the same exports. One major
> restriction is only one module with the same symbols can be loaded on the same
> time but it's ok in my case(as I described, in my case, it's per platform
> modules and devices with different hardware using the same software image).
>
> 3.
> >If there are multiple variants of the same module (again, is this about
> >external modules?), then I see no advantage to delay the decisions from
> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
>
> It can be different variant of the same module but maybe not. For example it
> can be cryptography modules. Modules provide the same api but implementation
> of api is totally different and depend on specific hardware. With modules
> alternatives feature it's easy to use this kind of modules. You can use
> required alternative for specific hardware and all depended modules can use
> external functions directly without any wrappers or "if" statements to resolve
> dependencies.
> With using depmod.d configuration it's possible to choose primary alternative
> into build time but in my case required alternative is unknown during build time,
> it will be known only into runtime. In this case it required to regenerate
> modules db into runtime and I try to avoid this.

It sounds to me, as if you would like to auto-generate modprobe.d/ files
for your platforms at boot time and implement the pre-loading of some
modules before some others with common modprobe.d syntax (e.g.
'install', cp. modprobe.d(5)).  But you probably evaluated that before
implementing your patches?

If your module load order is just platform dependent, I do not yet
understand, why (possibly boot-time dynamic) depmod.d configuration is
not sufficiently flexible enough.  I probably have missing some
important point.

I strongly support Lucas' request for a cover-letter.

Kind regards,
Nicolas

Re: [MODALTS v0.1 4/4] add modules deps alternatives description

[Lucas De Marchi]

On Mon, Jul 01, 2024 at 12:51:50PM GMT, Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco) wrote:
>Hi Nicolas,
>
>> It sounds to me, as if you would like to auto-generate modprobe.d/ files
>> for your platforms at boot time and implement the pre-loading of some
>> modules before some others with common modprobe.d syntax (e.g.
>> 'install', cp. modprobe.d(5)).  But you probably evaluated that before
>> implementing your patches?
>
>Some kind but not exactly. I try to provide method to avoid regenerating
>modules deps db during runtime
>
>> If your module load order is just platform dependent, I do not yet
>> understand, why (possibly boot-time dynamic) depmod.d configuration is
>> not sufficiently flexible enough.  I probably have missing some
>> important point.
>
>Device filesystem mounted as readonly to reduce security riscs so using
>depmod during boot time is restricted

depmod is never ever used during boot time... we should be able to boot
without even have depmod and even without modprobe (since libkmod used
by systemd should already have everything needed).

It looks like you have several modules exporting kind of duplicated
symbols and want something in runtime to disambiguate which module
should be loaded. I need an example to understand this... it's sounding
more like the chosen abstraction to have duplicated symbols (rather than a
xxx-common.ko module) is the main thing that could be changed without
adding complexity to module dep resolution.

Lucas De Marchi

>
>> I strongly support Lucas' request for a cover-letter.
>Probably, it will be a good idea to prepare test repo with couple of simple
>modules to demonstrate what patchset do
>
>> Kind regards,
>> Nicolas
>
>Thank you Nicolas for review :)
>Best regards,
>Valerii
>
>________________________________________
>From: Nicolas Schier <n.schier@avm.de>
>Sent: Monday, July 1, 2024 1:33 PM
>To: Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)
>Cc: Lucas De Marchi; linux-modules@vger.kernel.org; xe-linux-external(mailer list); Lucas De Marchi
>Subject: Re: [MODALTS v0.1 4/4] add modules deps alternatives description
>
>On Mon, Jul 01, 2024 at 09:23:03AM +0000, Valerii Chernous wrote:
>> >On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
>> >>Cc: xe-linux-external@cisco.com
>> >>Cc: Valerii Chernous <vchernou@cisco.com>
>> >>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
>> >>---
>> >> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
>> >> 1 file changed, 40 insertions(+)
>> >> create mode 100644 README.deps.alternatives.txt
>> >>
>> >>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
>> >>new file mode 100644
>> >>index 0000000..9ad3ce5
>> >>--- /dev/null
>> >>+++ b/README.deps.alternatives.txt
>> >>@@ -0,0 +1,40 @@
>> >>+Modules alternatives feature allow to calculate dependency alternatives
>> >>+during build time and aviod to regenerate modules db into runtime
>> >>+
>> >>+To enable deps alternatives calculation use "-D" flag with depmod,
>> >>+it will create indexes modules.alternatives and modules.alternatives.bin
>> >>+This indexes will be used by modprobe in runtime
>> >>+By default modprobe will load first(primary/major) dependency from list
>> >>+If it required to load alternative module, it should be done manually before
>> >>+loading main modules set.
>> >>+For example systemd service that detect platform type can load required platform
>> >>+modules and after it run main device initialization
>> >>+In case when alternative module loaded, modprobe detect this and skip to load primary
>> >>+dependency
>> >>+
>> >>+modules deps alternatives generation basic algorithm description
>> >>+1. Load modules information(imported/exported symbols)
>> >>+2. Find depended symbol alternatives(create list available symbols
>> >>+   alternatives instead of storing last one)
>> >>+3. Choise primary/major alternative per depended symbol correspond to
>> >>+   build time dependency(build time deps getting from module info section)
>> >>+4. Create a list of dependency modules alternatives correspond to next rule:
>> >>+4.1 All modules alternatives for module dependency should provide all symbols
>> >>+5 Store modules alternatives index(modules.alternatives) as key:value where
>> >>+key is a pair depended#_#primary_depency,
>> >>+value is list of all modules that provide all symbols from primary_depency
>> >>+for depended module
>> >>+
>> >>+Note:
>> >>+Current implementation/algorithm doesn't cover variant where requred symbols
>> >>+from primary deps provided by more that one modules. Exporting all symbols in
>> >>+alternative depency that used by depended module from primary_depency is
>> >>+mandatory!
>> >>+
>> >>+Note:
>> >>+modules.dep index different for standard/basic and modules alternatives algorithms
>> >>+modules.dep for modules alternatives algorithm contain only direct dependencies and
>> >>+full dependency list will be calculated into runtime correspond to preferred alternative.
>> >>+modules.dep for standard/basic algorithm contain full dependency list for module and
>> >>+can't be changed during runtime without rebuild database via depmod
>>
>>
>> >well... this kind of explains the what, but still no clue on why.
>> >If multiple different modules are providing the same symbol, then they
>> >are doing things wrong.
>>
>> >If there are multiple variants of the same module (again, is this about
>> >external modules?), then I see no advantage to delay the decisions from
>> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
>>
>> >Also end users have not visibility on a README.deps.alternatives.txt
>> >file. Documentation in kmod is kept on man pages.
>>
>>
>> >Lucas De Marchi
>>
>> First at all, thank you for review, Lucas.
>> Let me try to explain feature more:
>> 1. You are correct, feature tested on external modules
>>
>> 2.
>> >If multiple different modules are providing the same symbol, then they
>> >are doing things wrong.
>>
>> Modules exported the same api(the same functions) and on my opinion it's ok
>> and kernel process normally different modules with the same exports. One major
>> restriction is only one module with the same symbols can be loaded on the same
>> time but it's ok in my case(as I described, in my case, it's per platform
>> modules and devices with different hardware using the same software image).
>>
>> 3.
>> >If there are multiple variants of the same module (again, is this about
>> >external modules?), then I see no advantage to delay the decisions from
>> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
>>
>> It can be different variant of the same module but maybe not. For example it
>> can be cryptography modules. Modules provide the same api but implementation
>> of api is totally different and depend on specific hardware. With modules
>> alternatives feature it's easy to use this kind of modules. You can use
>> required alternative for specific hardware and all depended modules can use
>> external functions directly without any wrappers or "if" statements to resolve
>> dependencies.
>> With using depmod.d configuration it's possible to choose primary alternative
>> into build time but in my case required alternative is unknown during build time,
>> it will be known only into runtime. In this case it required to regenerate
>> modules db into runtime and I try to avoid this.
>
>It sounds to me, as if you would like to auto-generate modprobe.d/ files
>for your platforms at boot time and implement the pre-loading of some
>modules before some others with common modprobe.d syntax (e.g.
>'install', cp. modprobe.d(5)).  But you probably evaluated that before
>implementing your patches?
>
>If your module load order is just platform dependent, I do not yet
>understand, why (possibly boot-time dynamic) depmod.d configuration is
>not sufficiently flexible enough.  I probably have missing some
>important point.
>
>I strongly support Lucas' request for a cover-letter.
>
>Kind regards,
>Nicolas

Re: [MODALTS v0.1 4/4] add modules deps alternatives description

[Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)]

Hi Lucas, Nicolas

Sorry for late response. I was busy with other tasks.
I prepared repo with scripts and examples of kernel modules "Hello" like style
to demonstrate problem and how it can be solved via "modules alternatives".
Please, check my repo to see details:
https://github.com/Valery0xff/kmod_mod_alternatives_test


> depmod is never ever used during boot time... we should be able to boot
> without even have depmod and even without modprobe (since libkmod used
> by systemd should already have everything needed).
Modules alternatives feature is transparent for libkmod users(including
systemd) because it doesn't change kmod external api, only kmod internals
So systemd modules autoload service working well with this patchset

> It looks like you have several modules exporting kind of duplicated
> symbols and want something in runtime to disambiguate which module
> should be loaded. I need an example to understand this... it's sounding
Repo that I shared contain examples of modules to demonstrate problem

> more like the chosen abstraction to have duplicated symbols (rather than a
> xxx-common.ko module) is the main thing that could be changed without
> adding complexity to module dep resolution.
Yes, modules alternatives index allow to use modules with the same symbols but
from different providers and all deal goes smoothly, without any additional
configs, blacklist, changing index directory, all that required can be stored
into one location

If you don't want to do build kmod, kernel modules and run scripts you can
check deps_report.txt and load_mods_report.txt that presented into test repo

Best regards,
Valerii

________________________________________
From: Lucas De Marchi <lucas.demarchi@intel.com>
Sent: Wednesday, July 3, 2024 4:27 PM
To: Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)
Cc: Nicolas Schier; linux-modules@vger.kernel.org; xe-linux-external(mailer list); Lucas De Marchi
Subject: Re: [MODALTS v0.1 4/4] add modules deps alternatives description

On Mon, Jul 01, 2024 at 12:51:50PM GMT, Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco) wrote:
>Hi Nicolas,
>
>> It sounds to me, as if you would like to auto-generate modprobe.d/ files
>> for your platforms at boot time and implement the pre-loading of some
>> modules before some others with common modprobe.d syntax (e.g.
>> 'install', cp. modprobe.d(5)).  But you probably evaluated that before
>> implementing your patches?
>
>Some kind but not exactly. I try to provide method to avoid regenerating
>modules deps db during runtime
>
>> If your module load order is just platform dependent, I do not yet
>> understand, why (possibly boot-time dynamic) depmod.d configuration is
>> not sufficiently flexible enough.  I probably have missing some
>> important point.
>
>Device filesystem mounted as readonly to reduce security riscs so using
>depmod during boot time is restricted

depmod is never ever used during boot time... we should be able to boot
without even have depmod and even without modprobe (since libkmod used
by systemd should already have everything needed).

It looks like you have several modules exporting kind of duplicated
symbols and want something in runtime to disambiguate which module
should be loaded. I need an example to understand this... it's sounding
more like the chosen abstraction to have duplicated symbols (rather than a
xxx-common.ko module) is the main thing that could be changed without
adding complexity to module dep resolution.

Lucas De Marchi

>
>> I strongly support Lucas' request for a cover-letter.
>Probably, it will be a good idea to prepare test repo with couple of simple
>modules to demonstrate what patchset do
>
>> Kind regards,
>> Nicolas
>
>Thank you Nicolas for review :)
>Best regards,
>Valerii
>
>________________________________________
>From: Nicolas Schier <n.schier@avm.de>
>Sent: Monday, July 1, 2024 1:33 PM
>To: Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)
>Cc: Lucas De Marchi; linux-modules@vger.kernel.org; xe-linux-external(mailer list); Lucas De Marchi
>Subject: Re: [MODALTS v0.1 4/4] add modules deps alternatives description
>
>On Mon, Jul 01, 2024 at 09:23:03AM +0000, Valerii Chernous wrote:
>> >On Fri, May 10, 2024 at 04:21:28AM GMT, Valerii Chernous wrote:
>> >>Cc: xe-linux-external@cisco.com
>> >>Cc: Valerii Chernous <vchernou@cisco.com>
>> >>Signed-off-by: Valerii Chernous <vchernou@cisco.com>
>> >>---
>> >> README.deps.alternatives.txt | 40 ++++++++++++++++++++++++++++++++++++
>> >> 1 file changed, 40 insertions(+)
>> >> create mode 100644 README.deps.alternatives.txt
>> >>
>> >>diff --git a/README.deps.alternatives.txt b/README.deps.alternatives.txt
>> >>new file mode 100644
>> >>index 0000000..9ad3ce5
>> >>--- /dev/null
>> >>+++ b/README.deps.alternatives.txt
>> >>@@ -0,0 +1,40 @@
>> >>+Modules alternatives feature allow to calculate dependency alternatives
>> >>+during build time and aviod to regenerate modules db into runtime
>> >>+
>> >>+To enable deps alternatives calculation use "-D" flag with depmod,
>> >>+it will create indexes modules.alternatives and modules.alternatives.bin
>> >>+This indexes will be used by modprobe in runtime
>> >>+By default modprobe will load first(primary/major) dependency from list
>> >>+If it required to load alternative module, it should be done manually before
>> >>+loading main modules set.
>> >>+For example systemd service that detect platform type can load required platform
>> >>+modules and after it run main device initialization
>> >>+In case when alternative module loaded, modprobe detect this and skip to load primary
>> >>+dependency
>> >>+
>> >>+modules deps alternatives generation basic algorithm description
>> >>+1. Load modules information(imported/exported symbols)
>> >>+2. Find depended symbol alternatives(create list available symbols
>> >>+   alternatives instead of storing last one)
>> >>+3. Choise primary/major alternative per depended symbol correspond to
>> >>+   build time dependency(build time deps getting from module info section)
>> >>+4. Create a list of dependency modules alternatives correspond to next rule:
>> >>+4.1 All modules alternatives for module dependency should provide all symbols
>> >>+5 Store modules alternatives index(modules.alternatives) as key:value where
>> >>+key is a pair depended#_#primary_depency,
>> >>+value is list of all modules that provide all symbols from primary_depency
>> >>+for depended module
>> >>+
>> >>+Note:
>> >>+Current implementation/algorithm doesn't cover variant where requred symbols
>> >>+from primary deps provided by more that one modules. Exporting all symbols in
>> >>+alternative depency that used by depended module from primary_depency is
>> >>+mandatory!
>> >>+
>> >>+Note:
>> >>+modules.dep index different for standard/basic and modules alternatives algorithms
>> >>+modules.dep for modules alternatives algorithm contain only direct dependencies and
>> >>+full dependency list will be calculated into runtime correspond to preferred alternative.
>> >>+modules.dep for standard/basic algorithm contain full dependency list for module and
>> >>+can't be changed during runtime without rebuild database via depmod
>>
>>
>> >well... this kind of explains the what, but still no clue on why.
>> >If multiple different modules are providing the same symbol, then they
>> >are doing things wrong.
>>
>> >If there are multiple variants of the same module (again, is this about
>> >external modules?), then I see no advantage to delay the decisions from
>> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
>>
>> >Also end users have not visibility on a README.deps.alternatives.txt
>> >file. Documentation in kmod is kept on man pages.
>>
>>
>> >Lucas De Marchi
>>
>> First at all, thank you for review, Lucas.
>> Let me try to explain feature more:
>> 1. You are correct, feature tested on external modules
>>
>> 2.
>> >If multiple different modules are providing the same symbol, then they
>> >are doing things wrong.
>>
>> Modules exported the same api(the same functions) and on my opinion it's ok
>> and kernel process normally different modules with the same exports. One major
>> restriction is only one module with the same symbols can be loaded on the same
>> time but it's ok in my case(as I described, in my case, it's per platform
>> modules and devices with different hardware using the same software image).
>>
>> 3.
>> >If there are multiple variants of the same module (again, is this about
>> >external modules?), then I see no advantage to delay the decisions from
>> >depmod-time to modprobe-time. Just setup your depmod.d configuration.
>>
>> It can be different variant of the same module but maybe not. For example it
>> can be cryptography modules. Modules provide the same api but implementation
>> of api is totally different and depend on specific hardware. With modules
>> alternatives feature it's easy to use this kind of modules. You can use
>> required alternative for specific hardware and all depended modules can use
>> external functions directly without any wrappers or "if" statements to resolve
>> dependencies.
>> With using depmod.d configuration it's possible to choose primary alternative
>> into build time but in my case required alternative is unknown during build time,
>> it will be known only into runtime. In this case it required to regenerate
>> modules db into runtime and I try to avoid this.
>
>It sounds to me, as if you would like to auto-generate modprobe.d/ files
>for your platforms at boot time and implement the pre-loading of some
>modules before some others with common modprobe.d syntax (e.g.
>'install', cp. modprobe.d(5)).  But you probably evaluated that before
>implementing your patches?
>
>If your module load order is just platform dependent, I do not yet
>understand, why (possibly boot-time dynamic) depmod.d configuration is
>not sufficiently flexible enough.  I probably have missing some
>important point.
>
>I strongly support Lucas' request for a cover-letter.
>
>Kind regards,
>Nicolas