[Bug 219166] occasional block layer hang when setting 'echo noop > /sys/block/sda/queue/scheduler'

[bugzilla-daemon@kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=219166

--- Comment #13 from Richard W.M. Jones (rjones@redhat.com) ---
To document for myself and others what I did to reproduce the bug and get the
kernel stack trace ...

(1) libguestfs from git with this patch reverted:
https://github.com/libguestfs/libguestfs/commit/b2d682a4730ead8b4ae07e5aaf6fa230c5eec305

(2) Run guestfish in a loop until it hangs:

$ while LIBGUESTFS_BACKEND_SETTINGS=force_tcg ./run guestfish -a /dev/null run
-vx >& /tmp/log ; do echo -n . ; done


(3) Looking /tmp/log we can see it hung just after trying to set noop
scheduler:

$ tail -5 /tmp/log
+ echo 300
+ for f in /sys/block/sd*/device/timeout
+ echo 300
+ for f in /sys/block/{h,s,ub,v}d*/queue/scheduler
+ echo noop

(4) Check the log for the kernel version, install the corresponding kernel
debuginfo.

(5) Get virsh to produce a core dump of the VM:

$ virsh list 
 Id     Name                       State
--------------------------------------------
 1950   guestfs-lsdbxy71u4jg1w6x   running

$ virsh dump 1950 /var/tmp/core --memory-only

Domain '1950' dumped to /var/tmp/core

(6) Open in 'crash':

$ crash
/usr/lib/debug/lib/modules/6.11.0-0.rc5.20240830git20371ba12063.47.fc42.x86_64/vmlinux
/var/tmp/core

(7) List processes and find the one which hung:

crash> ps 
...
      230      73   0  ffffa01f83c58000  UN   0.3    11608     3340  modprobe

(8) Get stack trace from the hung process:

crash> set 230
    PID: 230
COMMAND: "modprobe"
   TASK: ffffa01f83c58000  [THREAD_INFO: ffffa01f83c58000]
    CPU: 0
  STATE: TASK_UNINTERRUPTIBLE 
crash> bt
PID: 230      TASK: ffffa01f83c58000  CPU: 0    COMMAND: "modprobe"
 #0 [ffffc1db0030f840] __schedule at ffffffff921906d0
 #1 [ffffc1db0030f8f8] schedule at ffffffff92191a27
 #2 [ffffc1db0030f908] __bio_queue_enter at ffffffff917e17dc
 #3 [ffffc1db0030f968] blk_mq_submit_bio at ffffffff917f3b4c
 #4 [ffffc1db0030f9f0] __submit_bio at ffffffff917e23fc
 #5 [ffffc1db0030fa58] submit_bio_noacct_nocheck at ffffffff917e28e6
 #6 [ffffc1db0030fac0] ext4_mpage_readpages at ffffffff915cef7c
 #7 [ffffc1db0030fbe0] read_pages at ffffffff91381cda
 #8 [ffffc1db0030fc40] page_cache_ra_unbounded at ffffffff91381fb5
 #9 [ffffc1db0030fca8] filemap_fault at ffffffff91376175
#10 [ffffc1db0030fd48] __do_fault at ffffffff913d1755
#11 [ffffc1db0030fd70] do_fault at ffffffff913d22e5
#12 [ffffc1db0030fda0] __handle_mm_fault at ffffffff913d8b2b
#13 [ffffc1db0030fe88] handle_mm_fault at ffffffff913d9472
#14 [ffffc1db0030fec8] do_user_addr_fault at ffffffff910b34ea
#15 [ffffc1db0030ff28] exc_page_fault at ffffffff92187e4e
#16 [ffffc1db0030ff50] asm_exc_page_fault at ffffffff922012a6
    RIP: 000055bb085508d8  RSP: 00007ffc3e731900  RFLAGS: 00010206
    RAX: 00000000000becd6  RBX: 00007f39925d1cd6  RCX: 000055bb0856592a
    RDX: 00007ffc3e733c70  RSI: 00000000200becd6  RDI: 000055bb1a712970
    RBP: 00007ffc3e731b60   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000202  R12: 00000000200becd6
    R13: 000055bb1a712970  R14: 00007ffc3e733c70  R15: 000055bb0856a708
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.