kprobe not always triggering

[Daniel Wagner <dwagner@suse.de>]

Hi,

it's likely a layer 8 problem but I can't figure out what I am doing
wrong.

I am debugging the nvme state machine with a nvme-tcp setup. I attached
a kprobe to nvme_change_ctrl_state to trace all the state changes, but
not all of them are logged. It's consistent behavior and I tried
different things with the same result.

The state transition I'd like to see is

  LIVE (1) -> RESETTING (2) -> CONNECTING (3) -> LIVE (1)

Though the 1 -> 2 transition is always missing, though a printk says it
is happening.

cd /sys/kernel/debug/tracing
echo "r:nvme_change_ctrl_state nvme_change_ctrl_state new_state=\$arg2:x32" > kprobe_events
echo 1 > events/kprobes/enable
echo 1 > tracing_on
cat trace_pipe

            nvme-570     [004] .....    97.559620: nvme_change_ctrl_state: (nvme_tcp_setup_ctrl+0x324/0x720 [nvme_tcp] <- nvme_change_ctrl_state) new_state=0x1
           <...>-65      [000] .....   143.540287: nvme_change_ctrl_state: (nvme_reset_ctrl_work+0x76/0xc0 [nvme_tcp] <- nvme_change_ctrl_state) new_state=0x3
   kworker/u34:3-76      [001] .....   170.236138: nvme_change_ctrl_state: (nvme_tcp_setup_ctrl+0x324/0x720 [nvme_tcp] <- nvme_change_ctrl_state) new_state=0x1


[   97.548827] nvme_change_ctrl_state:566 3 -> 1
[   97.550547] nvme nvme1: new ctrl: NQN "nqn.io-1", addr 192.168.154.25:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36
[  133.273552] nvme nvme1: I/O tag 1 (e001) type 4 opcode 0x18 (Keep Alive) QID 0 timeout
[  133.274359] nvme nvme1: long keepalive RTT (5056 ms)
[  133.274684] nvme nvme1: failed nvme_keep_alive_end_io error=10
[  143.514090] nvme_change_ctrl_state:566 1 -> 2
[  143.529185] nvme_change_ctrl_state:566 2 -> 3
[  146.584904] nvme nvme1: failed to connect socket: -110
[  146.585553] nvme nvme1: Reconnecting in 10 seconds...
[  159.897184] nvme nvme1: failed to connect socket: -110
[  159.897633] nvme nvme1: Failed reconnect attempt 2/60
[  159.898006] nvme nvme1: Reconnecting in 10 seconds...
[  170.153915] nvme nvme1: creating 8 I/O queues.

The system is idle, so I hope it's not just a dropped event. Any ideas
what I am doing wrong?

Linux 6.13.0-rc3+, clang compiler.

Thanks,
Daniel