[REGRESSION][BISECTED] Double energy consumption on idle

[REGRESSION][BISECTED] Double energy consumption on idle

[Kurt Borja]

Hi!

When I first booted into v6.13 I noticed my laptop got instantly hotter
and battery started draining fast. Today I bisected the kernel an ran
powerstat [1]. It comes down to

Upstream commit: 6b34562f0cfe ("crypto: akcipher - Drop sign/verify operations")

These results are reproducible on my system 100% of the time, and the
regression is still present on the latest upstream commit.

Please tell me if there is more info or test I should provide. I can
test any patch candidates too.

Happy holidays!

~ Kurt

[1] https://github.com/ColinIanKing/powerstat

Before commit 6b34562f0cfe
==========================

# powerstat -R
  Time    User  Nice   Sys  Idle    IO  Run Ctxt/s  IRQ/s Fork Exec Exit  Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
 Average   0.0   0.0   0.0  99.9   0.1  1.0 1926.9  335.5  0.1  0.0  0.2  24.94 
 GeoMean   0.0   0.0   0.0  99.9   0.0  1.0 1818.4  330.9  0.0  0.0  0.0  24.94 
  StdDev   0.0   0.0   0.0   0.0   0.0  0.0  650.8   60.1  0.2  0.0  1.1   0.21 
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
 Minimum   0.0   0.0   0.0  99.8   0.0  1.0  737.0  267.0  0.0  0.0  0.0  24.71 
 Maximum   0.2   0.0   0.1 100.0   0.1  1.0 3675.0  543.0  1.0  0.0  8.0  26.21 
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
Summary:
CPU:  24.94 Watts on average with standard deviation 0.21  
Note: power read from RAPL domains: uncore, pkg-0, core, psys.
These readings do not cover all the hardware in this device.

After commit 6b34562f0cfe
=========================

# powerstat -R
  Time    User  Nice   Sys  Idle    IO  Run Ctxt/s  IRQ/s Fork Exec Exit  Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
 Average   0.0   0.0   0.1  99.7   0.2  1.1 3280.0  408.1  0.3  0.1  0.4  57.64 
 GeoMean   0.0   0.0   0.0  99.7   0.1  1.0 2948.2  395.1  0.0  0.0  0.0  57.63 
  StdDev   0.0   0.0   0.1   0.2   0.1  0.3 1594.3  142.8  1.9  1.0  1.9   0.88 
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
 Minimum   0.0   0.0   0.0  98.3   0.1  1.0 1353.0  278.0  0.0  0.0  0.0  57.24 
 Maximum   0.3   0.1   0.5  99.9   0.9  3.0 8216.0 1386.0 14.0  8.0 14.0  64.24 
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ 
Summary:
CPU:  57.64 Watts on average with standard deviation 0.88  
Note: power read from RAPL domains: uncore, pkg-0, core, psys.
These readings do not cover all the hardware in this device.

Aditional Info
==============

# inxi -Fz
System:
  Kernel: 6.12.0-rc1-00010-ga16a17d3eaa4 arch: x86_64 bits: 64
  Desktop: KDE Plasma v: 6.2.4 Distro: Arch Linux
Machine:
  Type: Laptop System: Alienware product: Alienware x15 R1 v: 1.24.0
    serial: <superuser required>
  Mobo: Alienware model: Alienware x15 R1 v: A00
    serial: <superuser required> UEFI: Alienware v: 1.24.0 date: 08/08/2024
Battery:
  ID-1: BAT1 charge: 61.0 Wh (100.0%) condition: 61.0/87.0 Wh (70.2%)
CPU:
  Info: 8-core model: 11th Gen Intel Core i7-11800H bits: 64 type: MT MCP
    cache: L2: 10 MiB
  Speed (MHz): avg: 800 min/max: 800/4600 cores: 1: 800 2: 800 3: 800 4: 800
    5: 800 6: 800 7: 800 8: 800 9: 800 10: 800 11: 800 12: 800 13: 800 14: 800
    15: 800 16: 800
Graphics:
  Device-1: Intel TigerLake-H GT1 [UHD Graphics] driver: i915 v: kernel
  Device-2: NVIDIA GA104M [GeForce RTX 3070 Mobile / Max-Q] driver: nvidia
    v: 565.77
  Device-3: Microdia Integrated_Webcam_HD driver: uvcvideo type: USB
  Display: wayland server: X.org v: 1.21.1.15 with: Xwayland v: 24.1.4
    compositor: kwin_wayland driver: X: loaded: modesetting,nvidia dri: iris
    gpu: i915,nvidia resolution: 1536x864
  API: EGL v: 1.5 drivers: iris,nvidia
    platforms: gbm,wayland,x11,surfaceless,device
  API: OpenGL v: 4.6.0 compat-v: 4.6 vendor: intel mesa v: 24.3.2-arch1.1
    renderer: Mesa Intel UHD Graphics (TGL GT1)
  API: Vulkan v: 1.4.303 drivers: N/A surfaces: xcb,xlib,wayland
Audio:
  Device-1: Intel Tiger Lake-H HD Audio driver: sof-audio-pci-intel-tgl
  Device-2: NVIDIA GA104 High Definition Audio driver: snd_hda_intel
  API: ALSA v: k6.12.0-rc1-00010-ga16a17d3eaa4 status: kernel-api
  Server-1: PipeWire v: 1.2.7 status: active
Network:
  Device-1: Intel Wi-Fi 6E AX210/AX1675 2x2 [Typhoon Peak] driver: iwlwifi
  IF: wlp59s0 state: up mac: <filter>
  IF-ID-1: virbr0 state: down mac: <filter>
Bluetooth:
  Device-1: Intel AX210 Bluetooth driver: btusb type: USB
  Report: btmgmt ID: hci0 rfk-id: 0 state: down bt-service: disabled
    rfk-block: hardware: no software: yes address: N/A
RAID:
  Hardware-1: Intel Volume Management Device NVMe RAID Controller driver: vmd
Drives:
  Local Storage: total: 1.38 TiB used: 366.32 GiB (26.0%)
  ID-1: /dev/nvme0n1 vendor: SK Hynix model: PC711 NVMe 512GB
    size: 476.94 GiB
  ID-2: /dev/nvme1n1 vendor: Western Digital model: WD BLACK SN850X 1000GB
    size: 931.51 GiB
Partition:
  ID-1: / size: 914.81 GiB used: 365.67 GiB (40.0%) fs: ext4 dev: /dev/dm-0
  ID-2: /boot size: 1022 MiB used: 662.3 MiB (64.8%) fs: vfat
    dev: /dev/nvme1n1p1
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 49.0 C mobo: N/A sodimm: SODIMM C
  Fan Speeds (rpm): cpu: 2000
Info:
  Memory: total: 16 GiB note: est. available: 15.28 GiB used: 3.2 GiB (20.9%)
  Processes: 337 Uptime: 10m Shell: Zsh inxi: 3.3.36

Re: [REGRESSION][BISECTED] Double energy consumption on idle

[Lukas Wunner]

On Tue, Dec 24, 2024 at 07:42:49PM -0500, Kurt Borja wrote:
> When I first booted into v6.13 I noticed my laptop got instantly hotter
> and battery started draining fast. Today I bisected the kernel an ran
> powerstat [1]. It comes down to
> 
> Upstream commit: 6b34562f0cfe ("crypto: akcipher - Drop sign/verify operations")
> 
> These results are reproducible on my system 100% of the time, and the
> regression is still present on the latest upstream commit.
> 
> Please tell me if there is more info or test I should provide. I can
> test any patch candidates too.

The commit removes dead code and doesn't touch anything power-related.
So it's quite odd that it should cause a power regression.

Could you try reverting only this single commit and verify that the
issue goes away?  Just to double-check that this commit is really
the root cause.


> Before commit 6b34562f0cfe
> CPU:  24.94 Watts on average with standard deviation 0.21  
[...]
> After commit 6b34562f0cfe
> CPU:  57.64 Watts on average with standard deviation 0.88

That's a huge amount of extra power draw.  I'd suspect the Nvidia
discrete graphics to consume such an amount, so maybe it's not
powered down?  Have you enabled relevant Kconfig options such as
CONFIG_VGA_SWITCHEROO and CONFIG_DRM_NOUVEAU on the v6.13-rc kernel?

Thanks,

Lukas

Re: [REGRESSION][BISECTED] Double energy consumption on idle

[Lukas Wunner]

On Tue, Dec 24, 2024 at 07:42:49PM -0500, Kurt Borja wrote:
> When I first booted into v6.13 I noticed my laptop got instantly hotter
> and battery started draining fast. Today I bisected the kernel an ran
> powerstat [1]. It comes down to
> 
> Upstream commit: 6b34562f0cfe ("crypto: akcipher - Drop sign/verify operations")
[...]
> Graphics:
>   Device-1: Intel TigerLake-H GT1 [UHD Graphics] driver: i915 v: kernel
>   Device-2: NVIDIA GA104M [GeForce RTX 3070 Mobile / Max-Q] driver: nvidia
>     v: 565.77

I note that you're using the out-of-tree nvidia driver on v6.12.

The driver may be using the portions of the akcipher API that were
removed by the commit you bisected to.  E.g. this source file calls
crypto_akcipher_verify():

https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/libspdm_ecc.c

Are you seeing build or load errors for the nvidia module?

If the module is not loaded, no voltage / frequency scaling happens,
which would indeed result in the dGPU consuming an awful lot of power.

Thanks,

Lukas

Re: [REGRESSION][BISECTED] Double energy consumption on idle

[Kurt Borja]

On Wed, Dec 25, 2024 at 12:03:37PM +0100, Lukas Wunner wrote:
> On Tue, Dec 24, 2024 at 07:42:49PM -0500, Kurt Borja wrote:
> > When I first booted into v6.13 I noticed my laptop got instantly hotter
> > and battery started draining fast. Today I bisected the kernel an ran
> > powerstat [1]. It comes down to
> > 
> > Upstream commit: 6b34562f0cfe ("crypto: akcipher - Drop sign/verify operations")
> [...]
> > Graphics:
> >   Device-1: Intel TigerLake-H GT1 [UHD Graphics] driver: i915 v: kernel
> >   Device-2: NVIDIA GA104M [GeForce RTX 3070 Mobile / Max-Q] driver: nvidia
> >     v: 565.77
> 
> I note that you're using the out-of-tree nvidia driver on v6.12.
> 
> The driver may be using the portions of the akcipher API that were
> removed by the commit you bisected to.  E.g. this source file calls
> crypto_akcipher_verify():
> 
> https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/libspdm_ecc.c
> 
> Are you seeing build or load errors for the nvidia module?
> 
> If the module is not loaded, no voltage / frequency scaling happens,
> which would indeed result in the dGPU consuming an awful lot of power.

Hi Lukas,

This is indeed the case, 6b34562f0cfe breaks my nvidia module. I didn't
notice dkms was reporting errors when trying to install it. 

I'll look into it and report it to NVIDIA or my distro maintainers if
it's necessary.

Thank you so much for your help!

~ Kurt

> 
> Thanks,
> 
> Lukas

Re: [REGRESSION][BISECTED] Double energy consumption on idle

[Lukas Wunner]

On Wed, Dec 25, 2024 at 08:48:32AM -0500, Kurt Borja wrote:
> This is indeed the case, 6b34562f0cfe breaks my nvidia module. I didn't
> notice dkms was reporting errors when trying to install it. 
> 
> I'll look into it and report it to NVIDIA or my distro maintainers if
> it's necessary.

There's an issue open on GitHub:
https://github.com/NVIDIA/open-gpu-kernel-modules/issues/746

Nvidia folks commented there that they're already tracking this internally.

Thanks,

Lukas