Re: [PATCH] firmware_loader: Replace kmap() with kmap_local_page()

["Fabio M. De Francesco" <fmdefrancesco@gmail.com>]

On domenica 10 luglio 2022 13:57:34 CEST Greg Kroah-Hartman wrote:
> On Sun, Jul 10, 2022 at 01:18:16PM +0200, Fabio M. De Francesco wrote:
> > On domenica 10 luglio 2022 12:24:41 CEST Greg Kroah-Hartman wrote:
> > > On Sun, Jul 10, 2022 at 12:11:56PM +0200, Fabio M. De Francesco 
wrote:
> > > > The use of kmap() is being deprecated in favor of 
kmap_local_page().
> > > > 
> > > > With kmap_local_page() the mappings are per thread, CPU local, can 
take
> > > > page faults, and can be called from any context (including 
interrupts).
> > > 
> > > But that is not the case here for this kmap() instance?
> > 
> > I'm not 100% sure to get what you are asking here :-)
> > Probably you mean that kmap() can work here and you don't see reason 
for 
> > converting? Am I understanding correctly?
> 
> Yes, that is what I am saying, why is this conversion needed here?  A
> justification would be nice.
> 
> > OK, then...
> > 
> > kmap() is being deprecated in favor of kmap_local_page(). Please see 
> > highmem.rst which I have updated weeks ago (https://docs.kernel.org/vm/
> > highmem.html).
> > 
> > Two main problems with kmap(): (1) It comes with an overhead as mapping 
> > space is restricted and protected by a global lock for synchronization 
and 
> > (2) kmap() also requires global TLB invalidation when the kmap’s pool 
wraps 
> > and it might block when the mapping space is fully utilized until a 
slot 
> > becomes available.
> > 
> > kmap_local_page() should be preferred, where feasible, over all the 
others.
> 
> Ok, that is good to know, thanks for the pointer, you should put this in
> the changelog text for maintainers who did not know this (like myself)
> as it makes it easier to review.
> 
> > > If this is a
> > > simple search/replace, why is this not just done once and be done 
with
> > > it?
> > 
> > No, this job needs code inspection. After more than 25 conversions I 
can 
> > say that no more than 30% have been simple search and replace.
> > 
> > > > Call kmap_local_page() in firmware_loader wherever kmap() is 
currently
> > > > used. In firmware_rw() use the copy_{from,to}_page() helpers 
instead of
> > > > open coding the local mappings plus memcpy().
> > > 
> > > Isn't that just a different cleanup than the kmap() change?  Or is 
that
> > > tied to the fact that the other buffer is now allocated with
> > > kmap_local_page() instead of kmap()?
> > 
> > This kinds of changes have never been considered clean-ups by other 
> > maintainers (for an example please see commit e88a6a8fece9 ("binder: 
Use 
> > memcpy_{to,from}_page() in binder_alloc_do_buffer_copy()"). 
> > 
> > Using helpers has always been considered part of the conversions 
themselves 
> > and nobody has ever requested further patches for these.
> > 
> > > > Suggested-by: Ira Weiny <ira.weiny@intel.com>
> > > > Signed-off-by: Fabio M. De Francesco <fmdefrancesco@gmail.com>
> > > > ---
> > > >  drivers/base/firmware_loader/main.c  | 4 ++--
> > > >  drivers/base/firmware_loader/sysfs.c | 9 ++++-----
> > > >  2 files changed, 6 insertions(+), 7 deletions(-)
> > > 
> > > Did you run this through the firmware test framework?
> > 
> > No, sorry. I assumed (wrongly?) that this is one of those cases which 
don't 
> > need any tests. However I have nothing against testing. I've done them 
> > where they were absolutely needed for Btrfs conversions and kexec.
> 
> Running the kernel selftests for firmware would be great, please do so
> for your next version of this patch that fixes the
> ktest-robot-found-issues.
> 
> thanks,
> 
> greg k-h
> 
Greg,

According to your requests, I extended the changelog text adding those 
information about why kmap() should be avoided. Then I deleted that unused 
variable which I had overlooked and finally tested with firmware selftests.

I see that the outputs of selftests, regardless of running a 5.19.0-rc6  
kernel with or without my changes, show always the same error:

"not ok 1 selftests: firmware: fw_run_tests.sh # TIMEOUT 165 seconds".".

I ran those tests on a QEMU/KVM 32-bits VM, booting a vanilla 5.19.0-rc6 
kernel with HIGHMEM64GB enabled. 

As said, outputs don't change with or without my patch. Instead it changes 
with the latest openSUSE stock kernel (5.18.9-2-pae):

"ok 1 selftests: firmware: fw_run_tests.sh".

Unfortunately, I'm not familiar with kernel selftests. Any ideas about what 
could have made this tests fail? Is it expected? 

If not, I can try and figure out why these outputs are not what they should 
be (the second version of my patch can wait the time it takes).

Thanks,

Fabio

P.S.: Whoever wants to read the entire log of the selftests is welcome :-)

xp4ns3@tweed32:/usr/src/git/linux> sudo make -C tools/testing/selftests 
TARGETS=firmware run_tests
[sudo] password for root: 
make: Entering directory '/usr/src/git/linux/tools/testing/selftests'
make --no-builtin-rules ARCH=x86 -C ../../.. headers_install
make[1]: Entering directory '/usr/src/git/linux'
  INSTALL ./usr/include
make[1]: Leaving directory '/usr/src/git/linux'
make[1]: Entering directory '/usr/src/git/linux/tools/testing/selftests/
firmware'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/usr/src/git/linux/tools/testing/selftests/
firmware'
make[1]: Entering directory '/usr/src/git/linux/tools/testing/selftests/
firmware'
TAP version 13
1..1
# selftests: firmware: fw_run_tests.sh
# Running namespace test: 
# Testing with firmware in parent namespace (assumed to be same file system 
as PID1)
# Testing with firmware in child namespace
# OK
# -----------------------------------------------------
# Running kernel configuration test 1 -- rare
# Emulates:
# CONFIG_FW_LOADER=y
# CONFIG_FW_LOADER_USER_HELPER=n
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK=n
# ./fw_filesystem.sh: filesystem loading works
# ./fw_filesystem.sh: async filesystem loading works
# ./fw_filesystem.sh: platform loading works
# 
# Testing with the file present...
# Batched request_firmware() normal try #1: OK
# Batched request_firmware() normal try #2: OK
# Batched request_firmware() normal try #3: OK
# Batched request_firmware() normal try #4: OK
# Batched request_firmware() normal try #5: OK
# Batched request_firmware_into_buf() normal try #1: OK
# Batched request_firmware_into_buf() normal try #2: OK
# Batched request_firmware_into_buf() normal try #3: OK
# Batched request_firmware_into_buf() normal try #4: OK
# Batched request_firmware_into_buf() normal try #5: OK
# Batched request_firmware_direct() normal try #1: OK
# Batched request_firmware_direct() normal try #2: OK
# Batched request_firmware_direct() normal try #3: OK
# Batched request_firmware_direct() normal try #4: OK
# Batched request_firmware_direct() normal try #5: OK
# Batched request_firmware_nowait(uevent=true) normal try #1: OK
# Batched request_firmware_nowait(uevent=true) normal try #2: OK
# Batched request_firmware_nowait(uevent=true) normal try #3: OK
# Batched request_firmware_nowait(uevent=true) normal try #4: OK
# Batched request_firmware_nowait(uevent=true) normal try #5: OK
# Batched request_firmware_nowait(uevent=false) normal try #1: OK
# Batched request_firmware_nowait(uevent=false) normal try #2: OK
# Batched request_firmware_nowait(uevent=false) normal try #3: OK
# Batched request_firmware_nowait(uevent=false) normal try #4: OK
# Batched request_firmware_nowait(uevent=false) normal try #5: OK
# Test request_partial_firmware_into_buf() off=0 size=10: OK
# Test request_partial_firmware_into_buf() off=0 size=5: OK
# Test request_partial_firmware_into_buf() off=1 size=6: OK
# Test request_partial_firmware_into_buf() off=2 size=10: OK
# 
# Testing with the file missing...
# Batched request_firmware() nofile try #1: OK
# Batched request_firmware() nofile try #2: OK
# Batched request_firmware() nofile try #3: OK
# Batched request_firmware() nofile try #4: OK
# Batched request_firmware() nofile try #5: OK
# Batched request_firmware_into_buf() nofile try #1: OK
# Batched request_firmware_into_buf() nofile try #2: OK
# Batched request_firmware_into_buf() nofile try #3: OK
# Batched request_firmware_into_buf() nofile try #4: OK
# Batched request_firmware_into_buf() nofile try #5: OK
# Batched request_firmware_direct() nofile try #1: OK
# Batched request_firmware_direct() nofile try #2: OK
# Batched request_firmware_direct() nofile try #3: OK
# Batched request_firmware_direct() nofile try #4: OK
# Batched request_firmware_direct() nofile try #5: OK
# Batched request_firmware_nowait(uevent=true) nofile try #1: OK
# Batched request_firmware_nowait(uevent=true) nofile try #2: OK
# Batched request_firmware_nowait(uevent=true) nofile try #3: OK
# Batched request_firmware_nowait(uevent=true) nofile try #4: OK
# Batched request_firmware_nowait(uevent=true) nofile try #5: OK
# Batched request_firmware_nowait(uevent=false) nofile try #1: OK
# Batched request_firmware_nowait(uevent=false) nofile try #2: OK
# Batched request_firmware_nowait(uevent=false) nofile try #3: OK
# Batched request_firmware_nowait(uevent=false) nofile try #4: OK
# Batched request_firmware_nowait(uevent=false) nofile try #5: OK
# Test request_partial_firmware_into_buf() off=0 size=10 nofile: OK
# Test request_partial_firmware_into_buf() off=0 size=5 nofile: OK
# Test request_partial_firmware_into_buf() off=1 size=6 nofile: OK
# Test request_partial_firmware_into_buf() off=2 size=10 nofile: OK
# 
# Testing with both plain and XZ files present...
# Batched request_firmware() both try #1: OK
# Batched request_firmware() both try #2: OK
# Batched request_firmware() both try #3: OK
# Batched request_firmware() both try #4: OK
# Batched request_firmware() both try #5: OK
# Batched request_firmware_into_buf() both try #1: OK
# Batched request_firmware_into_buf() both try #2: OK
# Batched request_firmware_into_buf() both try #3: OK
# Batched request_firmware_into_buf() both try #4: OK
# Batched request_firmware_into_buf() both try #5: OK
# Batched request_firmware_direct() both try #1: OK
# Batched request_firmware_direct() both try #2: OK
# Batched request_firmware_direct() both try #3: OK
# Batched request_firmware_direct() both try #4: OK
# Batched request_firmware_direct() both try #5: OK
# Batched request_firmware_nowait(uevent=true) both try #1: OK
# Batched request_firmware_nowait(uevent=true) both try #2: OK
# Batched request_firmware_nowait(uevent=true) both try #3: OK
# Batched request_firmware_nowait(uevent=true) both try #4: OK
# Batched request_firmware_nowait(uevent=true) both try #5: OK
# Batched request_firmware_nowait(uevent=false) both try #1: OK
# Batched request_firmware_nowait(uevent=false) both try #2: OK
# Batched request_firmware_nowait(uevent=false) both try #3: OK
# Batched request_firmware_nowait(uevent=false) both try #4: OK
# Batched request_firmware_nowait(uevent=false) both try #5: OK
# 
# Testing with only XZ file present...
# Batched request_firmware() componly try #1: OK
# Batched request_firmware() componly try #2: OK
# Batched request_firmware() componly try #3: OK
# Batched request_firmware() componly try #4: OK
# Batched request_firmware() componly try #5: OK
# Batched request_firmware_into_buf() componly try #1: OK
# Batched request_firmware_into_buf() componly try #2: OK
# Batched request_firmware_into_buf() componly try #3: OK
# Batched request_firmware_into_buf() componly try #4: OK
# Batched request_firmware_into_buf() componly try #5: OK
# Batched request_firmware_direct() componly try #1: OK
# Batched request_firmware_direct() componly try #2: OK
# Batched request_firmware_direct() componly try #3: OK
# Batched request_firmware_direct() componly try #4: OK
# Batched request_firmware_direct() componly try #5: OK
# Batched request_firmware_nowait(uevent=true) componly try #1: OK
# Batched request_firmware_nowait(uevent=true) componly try #2: OK
# Batched request_firmware_nowait(uevent=true) componly try #3: OK
# Batched request_firmware_nowait(uevent=true) componly try #4: OK
# Batched request_firmware_nowait(uevent=true) componly try #5: OK
# Batched request_firmware_nowait(uevent=false) componly try #1: OK
# Batched request_firmware_nowait(uevent=false) componly try #2: OK
# Batched request_firmware_nowait(uevent=false) componly try #3: OK
# Batched request_firmware_nowait(uevent=false) componly try #4: OK
# Batched request_firmware_nowait(uevent=false) componly try #5: OK
# 
# Testing with both plain and ZSTD files present...
# Batched request_firmware() both try #1: OK
# Batched request_firmware() both try #2: OK
# Batched request_firmware() both try #3: OK
# Batched request_firmware() both try #4: OK
# Batched request_firmware() both try #5: OK
# Batched request_firmware_into_buf() both try #1: OK
# Batched request_firmware_into_buf() both try #2: OK
# Batched request_firmware_into_buf() both try #3: OK
# Batched request_firmware_into_buf() both try #4: OK
# Batched request_firmware_into_buf() both try #5: OK
# Batched request_firmware_direct() both try #1: OK
# Batched request_firmware_direct() both try #2: OK
# Batched request_firmware_direct() both try #3: OK
# Batched request_firmware_direct() both try #4: OK
# Batched request_firmware_direct() both try #5: OK
# Batched request_firmware_nowait(uevent=true) both try #1: OK
# Batched request_firmware_nowait(uevent=true) both try #2: OK
# Batched request_firmware_nowait(uevent=true) both try #3: OK
# Batched request_firmware_nowait(uevent=true) both try #4: OK
# Batched request_firmware_nowait(uevent=true) both try #5: OK
# Batched request_firmware_nowait(uevent=false) both try #1: OK
# Batched request_firmware_nowait(uevent=false) both try #2: OK
# Batched request_firmware_nowait(uevent=false) both try #3: OK
# Batched request_firmware_nowait(uevent=false) both try #4: OK
# Batched request_firmware_nowait(uevent=false) both try #5: OK
# 
# Testing with only ZSTD file present...
# Batched request_firmware() componly try #1: OK
# Batched request_firmware() componly try #2: OK
# Batched request_firmware() componly try #3: OK
# Batched request_firmware() componly try #4: OK
# Batched request_firmware() componly try #5: OK
# Batched request_firmware_into_buf() componly try #1: OK
# Batched request_firmware_into_buf() componly try #2: OK
# Batched request_firmware_into_buf() componly try #3: OK
# Batched request_firmware_into_buf() componly try #4: OK
# Batched request_firmware_into_buf() componly try #5: OK
# Batched request_firmware_direct() componly try #1: OK
# Batched request_firmware_direct() componly try #2: OK
# Batched request_firmware_direct() componly try #3: OK
# Batched request_firmware_direct() componly try #4: OK
# Batched request_firmware_direct() componly try #5: OK
# Batched request_firmware_nowait(uevent=true) componly try #1: OK
# Batched request_firmware_nowait(uevent=true) componly try #2: OK
# Batched request_firmware_nowait(uevent=true) componly try #3: OK
# Batched request_firmware_nowait(uevent=true) componly try #4: OK
# Batched request_firmware_nowait(uevent=true) componly try #5: OK
# Batched request_firmware_nowait(uevent=false) componly try #1: OK
# Batched request_firmware_nowait(uevent=false) componly try #2: OK
# Batched request_firmware_nowait(uevent=false) componly try #3: OK
# Batched request_firmware_nowait(uevent=false) componly try #4: OK
# Batched request_firmware_nowait(uevent=false) componly try #5: OK
#
not ok 1 selftests: firmware: fw_run_tests.sh # TIMEOUT 165 seconds
make[1]: Leaving directory '/usr/src/git/linux/tools/testing/selftests/
firmware'
make: Leaving directory '/usr/src/git/linux/tools/testing/selftests'