[ANNOUNCE] kmod 30

[ANNOUNCE] kmod 30

[Lucas De Marchi]

kmod 30 is out:

           https://www.kernel.org/pub/linux/utils/kernel/kmod/kmod-30.tar.xz
           https://www.kernel.org/pub/linux/utils/kernel/kmod/kmod-30.tar.sign

- Improvements
         - Stop adding duplicate information on modules.builtin.alias.bin, just use
           the modules.builtin.bin index

         - Speedup depmod, particularly under qemu with emulated arch, by
           avoiding a lot of open/read/close of modules.alias.bin. On an
           emulated ARM rootfs, depmod with only 2 modules was taking ~32s
           vs ~0.07s now.

         - Add kmod_module_new_from_name_lookup() which allows doing a lookup by
           module name, without considering the aliases. Other than that search
           order is similar to kmod_module_new_from_lookup().

         - modinfo learned the --modname option to explicitely show information
           about the module, even if there is an alias with the same name. This
           allows showing information about e.g. kernel/lib/crc32.ko, even if
           kernel also exports a crc32 alias in modules.alias:

                 alias crc32 crc32_pclmul
                 alias crc32 crc32_generic

           Same behavior will be used to other modules and to aliases provided
           by user/distro.

         - depmod.conf learned a new "excludedir" directive so distro/user can
           configure more directories to be excluded from its search, besides
           the hardcoded values "build" and "source".

         - Better group modprobe options on help output under "Management, Query and General".

         - modprobe learned a --wait <MSEC> option to be used together with -r
           when removing a module. This allows modprobe to keep trying the
           removal if it fails because the module is still in use. An exponential backoff
           time is used for further retries.

           The wait behavior provided by the kernel when not passing O_NONBLOCK
           to delete_module() was removed in v3.13 due to not be used and the
           consequences of having to support it in the kernel. However there may
           be some users, particularly on testsuites for individual susbsystems, that
           would want that. So provide a userspace implementation inside modprobe for
           such users. "rmmod" doesn't have a --wait as it remains a bare minimal over
           the API provided by the kernel. In future the --wait behavior can be added
           to libkmod for testsuites not exec'ing modprobe for module removal.

         - kmod_module_remove_module() learned a new flag to silence output when
           caller wants to handle them - this is particularly important for the
           --wait flag to modprobe, as it's not desired to keep seeing error messages
           while waiting for the module to be unused.

         - Add SM3 hash algo support to modinfo output, as already available in the kernel.

- Bug Fixes
         - Fix modinfo output when showing information for a .ko module when running
           on a kernel that has that module as builtin.

         - Fix kmod_module_new_from_lookup() returning > 0 rather than 0
           when it matches an alias.

         - Fix modinfo segfault when module doesn't exist.

         - Add missing function in the html documentation: kmod_get_dirname().

         - Fix modprobe incorrectly handling number of arguments when prepending values from
           MODPROBE_OPTIONS environment variable.

         - Fix modprobe -r --remove-dependencies and since "dependencies" was a
           misnomer, add the preferred argument option: "--remove-holders". This
           is the same name used by the kernel. It allows users to also remove
           other modules holding the one that is being removed.

         - Fix off-by-one in max module name length in depmod.

- Infra/internal
         - Start some changes in the out-of-tree test modules in kmod so they are useful
           for being really inserted in the kernel rather than relying on kmod's mock
           interface. This helps manual testing and may be used to exercise to test
           changes in the kernel.

Shortlog is below:

HuaxinLu (1):
       libkmod: Support SM3 hash algorithm

Lucas De Marchi (30):
       libkmod: Prefer builtin index over builtin.alias
       depmod: Do not duplicate builtin index
       depmod: Stop opening modules.modinfo once per module
       test-initstate: Check for negative value on error
       libkmod-module: Fix return code for kmod_module_new_from_lookup()
       gitignore: ignore gtk-doc.make
       libkmod: Add helper function to iterate lookup options
       libkmod: Update docs about indexes order
       libkmod: Add lookup from module name
       modinfo: Update help message with "modulename"
       modinfo: Allow to force arg as module name
       libkmod: Fix use of sizeof instead of ARRAY_SIZE
       docs: Add missing functions to documentation
       modprobe: Rename rmmod_do_deps_list
       modprobe: Fix holders removal
       modprobe: move check for remove_holders to caller
       modprobe: Make rmmod_do_module() contain all the removal sequence
       depmod: Fix writing over array length
       modprobe: re-use modname variable
       modprobe: Move -R to "Query options"
       libkmod: Allow to ignore log message on module removal
       module-playground: Add debugfs entry in mod-simple
       util: Add time-related functions from testsuite
       util: Add msec variants for time-related functions
       util: Add exponential backoff sleep
       testsuite: Add tests for sleep calculation
       modprobe: Add --wait
       Keep only one readme
       README: Update optional dependencies
       kmod 30

Luis Chamberlain (1):
       libkmod: add a library notice log level print

Masahiro Yamada (2):
       modprobe: fix the NULL-termination of new_argv
       modprobe: remove unneeded variable str_start

Michal Suchanek (1):
       libkmod: Set builtin to no when module is created from path.

Saul Wold (1):
       depmod: Add support for excluding a directory

Yauheni Kaliuta (1):
       libkmod-module: check "new_from_name" return value in get_builtin


Thank you all for the contributions.


Lucas De Marchi

Re: [ANNOUNCE] kmod 30

[Luis Chamberlain]

On Thu, Jun 30, 2022 at 08:36:21AM -0700, Lucas De Marchi wrote:
>         - modprobe learned a --wait <MSEC> option to be used together with -r
>           when removing a module. This allows modprobe to keep trying the
>           removal if it fails because the module is still in use. An exponential backoff
>           time is used for further retries.
> 
>           The wait behavior provided by the kernel when not passing O_NONBLOCK
>           to delete_module() was removed in v3.13 due to not be used and the
>           consequences of having to support it in the kernel. However there may
>           be some users, particularly on testsuites for individual susbsystems, that
>           would want that. So provide a userspace implementation inside modprobe for
>           such users. "rmmod" doesn't have a --wait as it remains a bare minimal over
>           the API provided by the kernel. In future the --wait behavior can be added
>           to libkmod for testsuites not exec'ing modprobe for module removal.

Sorry for the super late review, I was swamped. OK so the only issue
I can think of is that rmmod *used* to support the kernel wait support
with $(rmmod --wait) so wouldn't this be odd?

It is why I had gone with:

 -p | --remove-patiently   patiently removes the module
 -t | --timeout            timeout in ms to remove the module

You would know better though.

Also just curious, is it really terrible to just support waiting
forever?

  Luis

Re: [ANNOUNCE] kmod 30

[Lucas De Marchi]

On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
>On Thu, Jun 30, 2022 at 08:36:21AM -0700, Lucas De Marchi wrote:
>>         - modprobe learned a --wait <MSEC> option to be used together with -r
>>           when removing a module. This allows modprobe to keep trying the
>>           removal if it fails because the module is still in use. An exponential backoff
>>           time is used for further retries.
>>
>>           The wait behavior provided by the kernel when not passing O_NONBLOCK
>>           to delete_module() was removed in v3.13 due to not be used and the
>>           consequences of having to support it in the kernel. However there may
>>           be some users, particularly on testsuites for individual susbsystems, that
>>           would want that. So provide a userspace implementation inside modprobe for
>>           such users. "rmmod" doesn't have a --wait as it remains a bare minimal over
>>           the API provided by the kernel. In future the --wait behavior can be added
>>           to libkmod for testsuites not exec'ing modprobe for module removal.
>
>Sorry for the super late review, I was swamped. OK so the only issue
>I can think of is that rmmod *used* to support the kernel wait support
>with $(rmmod --wait) so wouldn't this be odd?

any reason not to use modprobe -r? Argument for rmmod supporting it in
the past is that the wait was implemented on the kernel side and rmmod
is the minimum wrapper around what the kernel provides.

On the other side, user shouldn't need to know where that is
implemented.

Over time libkmod grew much more to support loading/querying modules
rather than removing. I think for next version I will move some of the
module-removal support to libkmod rather than modprobe/rmmod. Then we
can think again on supporting that flag there.

>
>It is why I had gone with:
>
> -p | --remove-patiently   patiently removes the module
> -t | --timeout            timeout in ms to remove the module
>
>You would know better though.
>
>Also just curious, is it really terrible to just support waiting
>forever?

is there a use case for that? If we are trying to cover some races, I
imagine a small timeout would be sufficient. Also notice that if the
timeout is too big, so will be the interval between the retries.  On
your v2 I had suggested polling the refcnt so we would get notificed
on changes, but as you also noticed, that didn't work very well.  So I
went back to a time-based retry solution.

if there is a use-case, should we cap the interval between retries?

thanks
Lucas De Marchi

>
>  Luis

Re: [ANNOUNCE] kmod 30

[Luis Chamberlain]

On Thu, Jun 30, 2022 at 03:33:23PM -0700, Lucas De Marchi wrote:
> On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
> > Sorry for the super late review, I was swamped. OK so the only issue
> > I can think of is that rmmod *used* to support the kernel wait support
> > with $(rmmod --wait) so wouldn't this be odd?
> 
> any reason not to use modprobe -r? 

I was referring to old scripts which may have used $(rmmod --wait) before.
But since support for that was ripped, then yeah I can see that should
not be an issue.

However I can think of *one* issue, did we ever support `modprobe--wait`?

Because the way fstests / blktests would implement this feature
detection is with something like this now:

_has_modprobe_patient()
{
	modprobe --help >& /dev/null || return 1
	modprobe --help | grep -q -1 "remove-patiently" || return 1
	return 0
}

> > It is why I had gone with:
> > 
> > -p | --remove-patiently   patiently removes the module
> > -t | --timeout            timeout in ms to remove the module
> > 
> > You would know better though.
> > 
> > Also just curious, is it really terrible to just support waiting
> > forever?
> 
> is there a use case for that? If we are trying to cover some races, I
> imagine a small timeout would be sufficient. Also notice that if the
> timeout is too big, so will be the interval between the retries.  On
> your v2 I had suggested polling the refcnt so we would get notificed
> on changes, but as you also noticed, that didn't work very well.  So I
> went back to a time-based retry solution.
> 
> if there is a use-case, should we cap the interval between retries?

I really can't think of a use case except for catching glaring
unexpected bugs in test suites where the kernel developer would
really like to know something really bad happened, but even then
a timeout is likely desirable.

So just a heads up the timeout I'll use for fstests / blktests will be
of 100 seconds.

Thanks for this work!

  Luis

Re: [ANNOUNCE] kmod 30

[Lucas De Marchi]

On Fri, Jul 01, 2022 at 10:49:10AM -0700, Luis Chamberlain wrote:
>On Thu, Jun 30, 2022 at 03:33:23PM -0700, Lucas De Marchi wrote:
>> On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
>> > Sorry for the super late review, I was swamped. OK so the only issue
>> > I can think of is that rmmod *used* to support the kernel wait support
>> > with $(rmmod --wait) so wouldn't this be odd?
>>
>> any reason not to use modprobe -r?
>
>I was referring to old scripts which may have used $(rmmod --wait) before.
>But since support for that was ripped, then yeah I can see that should
>not be an issue.
>
>However I can think of *one* issue, did we ever support `modprobe--wait`?

no

>
>Because the way fstests / blktests would implement this feature
>detection is with something like this now:
>
>_has_modprobe_patient()
>{
>	modprobe --help >& /dev/null || return 1
>	modprobe --help | grep -q -1 "remove-patiently" || return 1
>	return 0
>}

the grep would need to be changed to something like "-w, --wait"

>
>> > It is why I had gone with:
>> >
>> > -p | --remove-patiently   patiently removes the module
>> > -t | --timeout            timeout in ms to remove the module
>> >
>> > You would know better though.
>> >
>> > Also just curious, is it really terrible to just support waiting
>> > forever?
>>
>> is there a use case for that? If we are trying to cover some races, I
>> imagine a small timeout would be sufficient. Also notice that if the
>> timeout is too big, so will be the interval between the retries.  On
>> your v2 I had suggested polling the refcnt so we would get notificed
>> on changes, but as you also noticed, that didn't work very well.  So I
>> went back to a time-based retry solution.
>>
>> if there is a use-case, should we cap the interval between retries?
>
>I really can't think of a use case except for catching glaring
>unexpected bugs in test suites where the kernel developer would
>really like to know something really bad happened, but even then
>a timeout is likely desirable.

yeah... if developer wants to possibly wait for a long time, `--wait -1`
is available to wait for years.

>
>So just a heads up the timeout I'll use for fstests / blktests will be
>of 100 seconds.

yeah... 100 seconds still make more sense than -1 IMO


Lucas De Marchi

>
>Thanks for this work!
>
>  Luis

Re: [ANNOUNCE] kmod 30

[Luis Chamberlain]

On Fri, Jul 01, 2022 at 11:13:21AM -0700, Lucas De Marchi wrote:
> On Fri, Jul 01, 2022 at 10:49:10AM -0700, Luis Chamberlain wrote:
> > On Thu, Jun 30, 2022 at 03:33:23PM -0700, Lucas De Marchi wrote:
> > > On Thu, Jun 30, 2022 at 03:09:32PM -0700, Luis Chamberlain wrote:
> > > > Sorry for the super late review, I was swamped. OK so the only issue
> > > > I can think of is that rmmod *used* to support the kernel wait support
> > > > with $(rmmod --wait) so wouldn't this be odd?
> > > 
> > > any reason not to use modprobe -r?
> > 
> > I was referring to old scripts which may have used $(rmmod --wait) before.
> > But since support for that was ripped, then yeah I can see that should
> > not be an issue.
> > 
> > However I can think of *one* issue, did we ever support `modprobe--wait`?
> 
> no
> 
> > 
> > Because the way fstests / blktests would implement this feature
> > detection is with something like this now:
> > 
> > _has_modprobe_patient()
> > {
> > 	modprobe --help >& /dev/null || return 1
> > 	modprobe --help | grep -q -1 "remove-patiently" || return 1
> > 	return 0
> > }
> 
> the grep would need to be changed to something like "-w, --wait"

Yes of course, that's easy, but we modprobe ever supportedl "--wait"
this would be a problem. Since it did not, we are good then!

> > > > It is why I had gone with:
> > > >
> > > > -p | --remove-patiently   patiently removes the module
> > > > -t | --timeout            timeout in ms to remove the module
> > > >
> > > > You would know better though.
> > > >
> > > > Also just curious, is it really terrible to just support waiting
> > > > forever?
> > > 
> > > is there a use case for that? If we are trying to cover some races, I
> > > imagine a small timeout would be sufficient. Also notice that if the
> > > timeout is too big, so will be the interval between the retries.  On
> > > your v2 I had suggested polling the refcnt so we would get notificed
> > > on changes, but as you also noticed, that didn't work very well.  So I
> > > went back to a time-based retry solution.
> > > 
> > > if there is a use-case, should we cap the interval between retries?
> > 
> > I really can't think of a use case except for catching glaring
> > unexpected bugs in test suites where the kernel developer would
> > really like to know something really bad happened, but even then
> > a timeout is likely desirable.
> 
> yeah... if developer wants to possibly wait for a long time, `--wait -1`
> is available to wait for years.

:) good call

> > So just a heads up the timeout I'll use for fstests / blktests will be
> > of 100 seconds.
> 
> yeah... 100 seconds still make more sense than -1 IMO

Great we'll go with that. Hopefully this will put a nail on the flaky
modules issue for good!

  Luis