* ocfs2 mount error @ 2024-03-10 21:46 Valentin Vidić 2024-03-11 8:37 ` Joseph Qi 0 siblings, 1 reply; 14+ messages in thread From: Valentin Vidić @ 2024-03-10 21:46 UTC (permalink / raw) To: ocfs2-devel, Alexander Aring, David Teigland, gfs2 Hi, I've started seeing the following mount error on recent kernels (6.7, 6.8) while the same script worked on older versions (6.1): 2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user 2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace 2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17 2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17 2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17 Script runs the commands in the following order: mkfs.ocfs2 fsck.ocfs2 mount.ocfs2 fsck command tries to release the lockspace but it seems to be busy so it stays around and the mount command fails. Running dlm_tool leave manually removes it and after that the mount command works again. A git bisect has led me to the following change: 61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks Could it be that some callback does not run correctly causing the lockspace to be busy? I have added some more debug info below including some logging of 5 kref_get and 4 kref_put that could be related. # dlm_tool ls dlm lockspaces name D6DCCE389FC84C8FB9E99ABE1D941825 id 0x3a105115 flags 0x00000000 change member 1 joined 1 remove 0 failed 0 seq 1,1 members 1 # dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825 cluster nodeid 1 quorate 1 ring seq 5 5 daemon now 1831 fence_pid 0 node 1 M add 54 rem 0 fail 0 fence 0 at 0 0 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags 2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1 2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,436839+00:00 Call Trace: 2024-03-10T21:04:29,437077+00:00 <TASK> 2024-03-10T21:04:29,437308+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,437653+00:00 add_lkb+0x2f/0x300 [dlm] 2024-03-10T21:04:29,438000+00:00 ? rsb_insert+0x85/0xd0 [dlm] 2024-03-10T21:04:29,438378+00:00 _grant_lock+0x111/0x170 [dlm] 2024-03-10T21:04:29,438756+00:00 do_request+0x71/0x90 [dlm] 2024-03-10T21:04:29,439112+00:00 _request_lock+0x19b/0x270 [dlm] 2024-03-10T21:04:29,439509+00:00 request_lock+0xc7/0x110 [dlm] 2024-03-10T21:04:29,439887+00:00 dlm_user_request+0x10d/0x370 [dlm] 2024-03-10T21:04:29,440304+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,440763+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,441161+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,441578+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,442016+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,442413+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,442723+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,443082+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,443389+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,443729+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,444086+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1 2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,455462+00:00 Call Trace: 2024-03-10T21:04:29,455922+00:00 <TASK> 2024-03-10T21:04:29,456345+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,456903+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] 2024-03-10T21:04:29,457548+00:00 do_request+0x7b/0x90 [dlm] 2024-03-10T21:04:29,458128+00:00 _request_lock+0x19b/0x270 [dlm] 2024-03-10T21:04:29,458771+00:00 request_lock+0xc7/0x110 [dlm] 2024-03-10T21:04:29,459375+00:00 dlm_user_request+0x10d/0x370 [dlm] 2024-03-10T21:04:29,460021+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,460714+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,461363+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,462003+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,462659+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,463268+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,463791+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,464359+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,464891+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,465440+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,466005+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1 2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,477545+00:00 Call Trace: 2024-03-10T21:04:29,478031+00:00 <TASK> 2024-03-10T21:04:29,478494+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,479094+00:00 dlm_user_request+0x299/0x370 [dlm] 2024-03-10T21:04:29,479787+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,480544+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,481195+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,481848+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,482518+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,483140+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,483672+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,484254+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,484785+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,485380+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,485954+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1 2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,419629+00:00 Call Trace: 2024-03-10T21:04:30,420115+00:00 <TASK> 2024-03-10T21:04:30,420566+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,421157+00:00 find_lkb+0x4e/0xd0 [dlm] 2024-03-10T21:04:30,421758+00:00 dlm_user_unlock+0x6b/0x2f0 [dlm] 2024-03-10T21:04:30,422422+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,423140+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,423785+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,424404+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,425057+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,425606+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,426268+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,426857+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,427415+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,427969+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,428534+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,429098+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,429657+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,430211+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1 2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,443065+00:00 Call Trace: 2024-03-10T21:04:30,443571+00:00 <TASK> 2024-03-10T21:04:30,444029+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,444618+00:00 _unlock_lock+0x73/0x110 [dlm] 2024-03-10T21:04:30,445257+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,445979+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,446641+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,447345+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,447979+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,448593+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,449460+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,450394+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,451160+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,451773+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,452316+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,452894+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,453491+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,454053+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,454621+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,455175+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1 2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,466737+00:00 Call Trace: 2024-03-10T21:04:30,467220+00:00 <TASK> 2024-03-10T21:04:30,467684+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,468276+00:00 _unlock_lock+0x9e/0x110 [dlm] 2024-03-10T21:04:30,468912+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,469607+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,470261+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,470961+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,471611+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,472221+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,472867+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,473397+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,474055+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,474633+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,475179+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,475759+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,476322+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,476883+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,477442+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,478000+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1 2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,489384+00:00 Call Trace: 2024-03-10T21:04:30,489869+00:00 <TASK> 2024-03-10T21:04:30,490320+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,490918+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] 2024-03-10T21:04:30,491609+00:00 _unlock_lock+0xbe/0x110 [dlm] 2024-03-10T21:04:30,492231+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,492914+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,493553+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,494238+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,494869+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,495479+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,496133+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,496660+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,497318+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,497898+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,498428+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,498978+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,499532+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,500086+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,500639+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,501192+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1 2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,512564+00:00 Call Trace: 2024-03-10T21:04:30,513045+00:00 <TASK> 2024-03-10T21:04:30,513496+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,514081+00:00 __put_lkb+0x30/0x100 [dlm] 2024-03-10T21:04:30,514683+00:00 dlm_user_add_ast+0x1cc/0x300 [dlm] 2024-03-10T21:04:30,515337+00:00 _unlock_lock+0xbe/0x110 [dlm] 2024-03-10T21:04:30,515953+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,516633+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,517265+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,517957+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,518586+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,519199+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,519840+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,520365+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,521021+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,521586+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,522100+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,522654+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,523212+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,523779+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,524333+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,524888+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1 2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,536247+00:00 Call Trace: 2024-03-10T21:04:30,536729+00:00 <TASK> 2024-03-10T21:04:30,537179+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,537755+00:00 __put_lkb+0x30/0x100 [dlm] 2024-03-10T21:04:30,538355+00:00 dlm_user_unlock+0x213/0x2f0 [dlm] 2024-03-10T21:04:30,539003+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,539698+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,540319+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,540927+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,541569+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,542093+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,542759+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,543327+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,543851+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,544395+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,544949+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,545500+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,546059+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,546621+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0 -- Valentin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-10 21:46 ocfs2 mount error Valentin Vidić @ 2024-03-11 8:37 ` Joseph Qi 2024-03-11 9:02 ` Valentin Vidić 0 siblings, 1 reply; 14+ messages in thread From: Joseph Qi @ 2024-03-11 8:37 UTC (permalink / raw) To: Valentin Vidić, ocfs2-devel, Alexander Aring, David Teigland, gfs2, ocfs2-devel Hi, It seems that you are using fsdlm. But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it? Thanks, Joseph On 3/11/24 5:46 AM, Valentin Vidić wrote: > Hi, > > I've started seeing the following mount error on recent kernels (6.7, 6.8) > while the same script worked on older versions (6.1): > > 2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user > 2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace > 2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17 > 2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17 > 2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17 > > Script runs the commands in the following order: > > mkfs.ocfs2 > fsck.ocfs2 > mount.ocfs2 > > fsck command tries to release the lockspace but it seems to be busy so > it stays around and the mount command fails. Running dlm_tool leave > manually removes it and after that the mount command works again. > > A git bisect has led me to the following change: > > 61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks > > Could it be that some callback does not run correctly causing the lockspace > to be busy? I have added some more debug info below including some logging of > 5 kref_get and 4 kref_put that could be related. > > # dlm_tool ls > dlm lockspaces > name D6DCCE389FC84C8FB9E99ABE1D941825 > id 0x3a105115 > flags 0x00000000 > change member 1 joined 1 remove 0 failed 0 seq 1,1 > members 1 > > # dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825 > cluster nodeid 1 quorate 1 ring seq 5 5 > daemon now 1831 fence_pid 0 > node 1 M add 54 rem 0 fail 0 fence 0 at 0 0 > > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000" > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21" > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda" > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda" > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name > /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags > > 2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1 > 2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:29,436839+00:00 Call Trace: > 2024-03-10T21:04:29,437077+00:00 <TASK> > 2024-03-10T21:04:29,437308+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:29,437653+00:00 add_lkb+0x2f/0x300 [dlm] > 2024-03-10T21:04:29,438000+00:00 ? rsb_insert+0x85/0xd0 [dlm] > 2024-03-10T21:04:29,438378+00:00 _grant_lock+0x111/0x170 [dlm] > 2024-03-10T21:04:29,438756+00:00 do_request+0x71/0x90 [dlm] > 2024-03-10T21:04:29,439112+00:00 _request_lock+0x19b/0x270 [dlm] > 2024-03-10T21:04:29,439509+00:00 request_lock+0xc7/0x110 [dlm] > 2024-03-10T21:04:29,439887+00:00 dlm_user_request+0x10d/0x370 [dlm] > 2024-03-10T21:04:29,440304+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:29,440763+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:29,441161+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] > 2024-03-10T21:04:29,441578+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] > 2024-03-10T21:04:29,442016+00:00 device_write+0x5f1/0x790 [dlm] > 2024-03-10T21:04:29,442413+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:29,442723+00:00 ? __x64_sys_fcntl+0x90/0xc0 > 2024-03-10T21:04:29,443082+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:29,443389+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:29,443729+00:00 ? exc_page_fault+0x73/0x170 > 2024-03-10T21:04:29,444086+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1 > 2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:29,455462+00:00 Call Trace: > 2024-03-10T21:04:29,455922+00:00 <TASK> > 2024-03-10T21:04:29,456345+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:29,456903+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] > 2024-03-10T21:04:29,457548+00:00 do_request+0x7b/0x90 [dlm] > 2024-03-10T21:04:29,458128+00:00 _request_lock+0x19b/0x270 [dlm] > 2024-03-10T21:04:29,458771+00:00 request_lock+0xc7/0x110 [dlm] > 2024-03-10T21:04:29,459375+00:00 dlm_user_request+0x10d/0x370 [dlm] > 2024-03-10T21:04:29,460021+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:29,460714+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:29,461363+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] > 2024-03-10T21:04:29,462003+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] > 2024-03-10T21:04:29,462659+00:00 device_write+0x5f1/0x790 [dlm] > 2024-03-10T21:04:29,463268+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:29,463791+00:00 ? __x64_sys_fcntl+0x90/0xc0 > 2024-03-10T21:04:29,464359+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:29,464891+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:29,465440+00:00 ? exc_page_fault+0x73/0x170 > 2024-03-10T21:04:29,466005+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1 > 2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:29,477545+00:00 Call Trace: > 2024-03-10T21:04:29,478031+00:00 <TASK> > 2024-03-10T21:04:29,478494+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:29,479094+00:00 dlm_user_request+0x299/0x370 [dlm] > 2024-03-10T21:04:29,479787+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:29,480544+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:29,481195+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] > 2024-03-10T21:04:29,481848+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] > 2024-03-10T21:04:29,482518+00:00 device_write+0x5f1/0x790 [dlm] > 2024-03-10T21:04:29,483140+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:29,483672+00:00 ? __x64_sys_fcntl+0x90/0xc0 > 2024-03-10T21:04:29,484254+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:29,484785+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:29,485380+00:00 ? exc_page_fault+0x73/0x170 > 2024-03-10T21:04:29,485954+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1 > 2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:30,419629+00:00 Call Trace: > 2024-03-10T21:04:30,420115+00:00 <TASK> > 2024-03-10T21:04:30,420566+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:30,421157+00:00 find_lkb+0x4e/0xd0 [dlm] > 2024-03-10T21:04:30,421758+00:00 dlm_user_unlock+0x6b/0x2f0 [dlm] > 2024-03-10T21:04:30,422422+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:30,423140+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:30,423785+00:00 device_write+0x5bd/0x790 [dlm] > 2024-03-10T21:04:30,424404+00:00 ? security_file_permission+0x32/0x50 > 2024-03-10T21:04:30,425057+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:30,425606+00:00 ? syscall_exit_to_user_mode+0x81/0x220 > 2024-03-10T21:04:30,426268+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,426857+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:30,427415+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:30,427969+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,428534+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,429098+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,429657+00:00 ? __irq_exit_rcu+0x3b/0xb0 > 2024-03-10T21:04:30,430211+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1 > 2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:30,443065+00:00 Call Trace: > 2024-03-10T21:04:30,443571+00:00 <TASK> > 2024-03-10T21:04:30,444029+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:30,444618+00:00 _unlock_lock+0x73/0x110 [dlm] > 2024-03-10T21:04:30,445257+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] > 2024-03-10T21:04:30,445979+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] > 2024-03-10T21:04:30,446641+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:30,447345+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:30,447979+00:00 device_write+0x5bd/0x790 [dlm] > 2024-03-10T21:04:30,448593+00:00 ? security_file_permission+0x32/0x50 > 2024-03-10T21:04:30,449460+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:30,450394+00:00 ? syscall_exit_to_user_mode+0x81/0x220 > 2024-03-10T21:04:30,451160+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,451773+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:30,452316+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:30,452894+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,453491+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,454053+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,454621+00:00 ? __irq_exit_rcu+0x3b/0xb0 > 2024-03-10T21:04:30,455175+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1 > 2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:30,466737+00:00 Call Trace: > 2024-03-10T21:04:30,467220+00:00 <TASK> > 2024-03-10T21:04:30,467684+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:30,468276+00:00 _unlock_lock+0x9e/0x110 [dlm] > 2024-03-10T21:04:30,468912+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] > 2024-03-10T21:04:30,469607+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] > 2024-03-10T21:04:30,470261+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:30,470961+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:30,471611+00:00 device_write+0x5bd/0x790 [dlm] > 2024-03-10T21:04:30,472221+00:00 ? security_file_permission+0x32/0x50 > 2024-03-10T21:04:30,472867+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:30,473397+00:00 ? syscall_exit_to_user_mode+0x81/0x220 > 2024-03-10T21:04:30,474055+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,474633+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:30,475179+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:30,475759+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,476322+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,476883+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,477442+00:00 ? __irq_exit_rcu+0x3b/0xb0 > 2024-03-10T21:04:30,478000+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1 > 2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:30,489384+00:00 Call Trace: > 2024-03-10T21:04:30,489869+00:00 <TASK> > 2024-03-10T21:04:30,490320+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:30,490918+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] > 2024-03-10T21:04:30,491609+00:00 _unlock_lock+0xbe/0x110 [dlm] > 2024-03-10T21:04:30,492231+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] > 2024-03-10T21:04:30,492914+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] > 2024-03-10T21:04:30,493553+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:30,494238+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:30,494869+00:00 device_write+0x5bd/0x790 [dlm] > 2024-03-10T21:04:30,495479+00:00 ? security_file_permission+0x32/0x50 > 2024-03-10T21:04:30,496133+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:30,496660+00:00 ? syscall_exit_to_user_mode+0x81/0x220 > 2024-03-10T21:04:30,497318+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,497898+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:30,498428+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:30,498978+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,499532+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,500086+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,500639+00:00 ? __irq_exit_rcu+0x3b/0xb0 > 2024-03-10T21:04:30,501192+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1 > 2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:30,512564+00:00 Call Trace: > 2024-03-10T21:04:30,513045+00:00 <TASK> > 2024-03-10T21:04:30,513496+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:30,514081+00:00 __put_lkb+0x30/0x100 [dlm] > 2024-03-10T21:04:30,514683+00:00 dlm_user_add_ast+0x1cc/0x300 [dlm] > 2024-03-10T21:04:30,515337+00:00 _unlock_lock+0xbe/0x110 [dlm] > 2024-03-10T21:04:30,515953+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] > 2024-03-10T21:04:30,516633+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] > 2024-03-10T21:04:30,517265+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:30,517957+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:30,518586+00:00 device_write+0x5bd/0x790 [dlm] > 2024-03-10T21:04:30,519199+00:00 ? security_file_permission+0x32/0x50 > 2024-03-10T21:04:30,519840+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:30,520365+00:00 ? syscall_exit_to_user_mode+0x81/0x220 > 2024-03-10T21:04:30,521021+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,521586+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:30,522100+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:30,522654+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,523212+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,523779+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,524333+00:00 ? __irq_exit_rcu+0x3b/0xb0 > 2024-03-10T21:04:30,524888+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > > 2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1 > 2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 > 2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > 2024-03-10T21:04:30,536247+00:00 Call Trace: > 2024-03-10T21:04:30,536729+00:00 <TASK> > 2024-03-10T21:04:30,537179+00:00 dump_stack_lvl+0x43/0x60 > 2024-03-10T21:04:30,537755+00:00 __put_lkb+0x30/0x100 [dlm] > 2024-03-10T21:04:30,538355+00:00 dlm_user_unlock+0x213/0x2f0 [dlm] > 2024-03-10T21:04:30,539003+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 > 2024-03-10T21:04:30,539698+00:00 ? device_write+0x44/0x790 [dlm] > 2024-03-10T21:04:30,540319+00:00 device_write+0x5bd/0x790 [dlm] > 2024-03-10T21:04:30,540927+00:00 ? security_file_permission+0x32/0x50 > 2024-03-10T21:04:30,541569+00:00 vfs_write+0xec/0x470 > 2024-03-10T21:04:30,542093+00:00 ? syscall_exit_to_user_mode+0x81/0x220 > 2024-03-10T21:04:30,542759+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,543327+00:00 ksys_write+0x6b/0xf0 > 2024-03-10T21:04:30,543851+00:00 do_syscall_64+0x87/0x1a0 > 2024-03-10T21:04:30,544395+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,544949+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,545500+00:00 ? do_syscall_64+0x93/0x1a0 > 2024-03-10T21:04:30,546059+00:00 ? __irq_exit_rcu+0x3b/0xb0 > 2024-03-10T21:04:30,546621+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 > 2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0 > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 8:37 ` Joseph Qi @ 2024-03-11 9:02 ` Valentin Vidić 2024-03-11 9:27 ` Joseph Qi 2024-03-11 15:20 ` Alexander Aring 0 siblings, 2 replies; 14+ messages in thread From: Valentin Vidić @ 2024-03-11 9:02 UTC (permalink / raw) To: Joseph Qi; +Cc: ocfs2-devel, Alexander Aring, David Teigland, gfs2, ocfs2-devel On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote: > It seems that you are using fsdlm. > But from your running commands, I don't see how fsdlm get involved. > Could you please give more detail on how to reproduce it? Sure, this is the original Pacemaker test script that failed: https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads Basically it starts Corosync and DLM services and than calls: mkfs.ocfs2 fsck.ocfs2 mount.ocfs2 -- Valentin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 9:02 ` Valentin Vidić @ 2024-03-11 9:27 ` Joseph Qi 2024-03-11 12:28 ` Heming Zhao 2024-03-11 15:20 ` Alexander Aring 1 sibling, 1 reply; 14+ messages in thread From: Joseph Qi @ 2024-03-11 9:27 UTC (permalink / raw) To: Valentin Vidić Cc: Alexander Aring, David Teigland, gfs2, ocfs2-devel, Heming Zhao On 3/11/24 5:02 PM, Valentin Vidić wrote: > On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote: >> It seems that you are using fsdlm. >> But from your running commands, I don't see how fsdlm get involved. >> Could you please give more detail on how to reproduce it? > > Sure, this is the original Pacemaker test script that failed: > > https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads > > Basically it starts Corosync and DLM services and than calls: > > mkfs.ocfs2 > fsck.ocfs2 > mount.ocfs2 > Thanks for quick reply. Now it confirms that it is using fsdlm. Joseph ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 9:27 ` Joseph Qi @ 2024-03-11 12:28 ` Heming Zhao 0 siblings, 0 replies; 14+ messages in thread From: Heming Zhao @ 2024-03-11 12:28 UTC (permalink / raw) To: Joseph Qi, Valentin Vidić Cc: Alexander Aring, David Teigland, gfs2, ocfs2-devel On 3/11/24 17:27, Joseph Qi wrote: > > > On 3/11/24 5:02 PM, Valentin Vidić wrote: >> On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote: >>> It seems that you are using fsdlm. >>> But from your running commands, I don't see how fsdlm get involved. >>> Could you please give more detail on how to reproduce it? >> >> Sure, this is the original Pacemaker test script that failed: >> >> https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads >> >> Basically it starts Corosync and DLM services and than calls: >> >> mkfs.ocfs2 >> fsck.ocfs2 >> mount.ocfs2 >> > > Thanks for quick reply. > Now it confirms that it is using fsdlm. > > Joseph > > It looks like I met the same issue one year ago. check [1] and search for "error 17." in this mail. [1]: https://lore.kernel.org/ocfs2-devel/20230626150916.je3egonzb3crvbl5@p15/ If my memory is still right, this root cause of this issue is that fsck forgets to clear the lockspace before exiting. I will investigate the debian's issue and provide a fix. Thanks, Heming ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 9:02 ` Valentin Vidić 2024-03-11 9:27 ` Joseph Qi @ 2024-03-11 15:20 ` Alexander Aring 2024-03-11 19:27 ` Valentin Vidić 1 sibling, 1 reply; 14+ messages in thread From: Alexander Aring @ 2024-03-11 15:20 UTC (permalink / raw) To: Valentin Vidić Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel Hi, On Mon, Mar 11, 2024 at 5:02 AM Valentin Vidić <vvidic@valentin-vidic.from.hr> wrote: > > On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote: > > It seems that you are using fsdlm. > > But from your running commands, I don't see how fsdlm get involved. > > Could you please give more detail on how to reproduce it? > > Sure, this is the original Pacemaker test script that failed: > > https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads > > Basically it starts Corosync and DLM services and than calls: > > mkfs.ocfs2 > fsck.ocfs2 when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me: fsck.ocfs2: Unable to access cluster service while initializing the DLM Looking into strace right before that I see a: connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = -1 ECONNREFUSED (Connection refused) it's true I don't have ocfs2_controld running, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions. Maybe somebody else can help so I can try to reproduce it? btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the device() read for ast callbacks are there. - Alex ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 15:20 ` Alexander Aring @ 2024-03-11 19:27 ` Valentin Vidić 2024-03-11 23:43 ` Alexander Aring 0 siblings, 1 reply; 14+ messages in thread From: Valentin Vidić @ 2024-03-11 19:27 UTC (permalink / raw) To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel On Mon, Mar 11, 2024 at 11:20:56AM -0400, Alexander Aring wrote: > when I try doing your steps (btw: thanks for sharing) to run > fsck.ocfs2 it tells me: > > fsck.ocfs2: Unable to access cluster service while initializing the DLM > > Looking into strace right before that I see a: > > connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = > -1 ECONNREFUSED (Connection refused) > > it's true I don't have ocfs2_controld running, after looking into > ocfs2-tools source code, I declare it as not compilable with recent > corosync versions. Not sure about that, because I also don't have ocfs2_controld running. This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack should be set with: echo pcmk > /sys/fs/ocfs2/cluster_stack > btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. > After I look at your debugging information it would be nice to see if > the device() read for ast callbacks are there. Here is a trace report for the fsck call: cpus=1 fsck.ocfs2-1696 [000] 54.621075: dlm_lock_start: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 54.692487: dlm_lock_end: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 54.695447: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 54.695636: dlm_lock_start: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 54.766959: dlm_lock_end: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 54.770765: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 54.770821: dlm_lock_start: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.845613: dlm_lock_end: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.848395: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.868951: dlm_unlock_start: ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.958319: dlm_unlock_end: ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.982397: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.982450: dlm_lock_start: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.056390: dlm_lock_end: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.058946: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.082546: dlm_unlock_start: ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.182903: dlm_unlock_end: ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.210128: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.530687: dlm_unlock_start: ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 55.625015: dlm_unlock_end: ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 55.650054: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 55.672033: dlm_unlock_start: ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 55.766844: dlm_unlock_end: ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 55.791829: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331 -- Valentin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 19:27 ` Valentin Vidić @ 2024-03-11 23:43 ` Alexander Aring 2024-03-12 8:09 ` Valentin Vidić 0 siblings, 1 reply; 14+ messages in thread From: Alexander Aring @ 2024-03-11 23:43 UTC (permalink / raw) To: Valentin Vidić Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel Hi, On Mon, Mar 11, 2024 at 3:27 PM Valentin Vidić <vvidic@valentin-vidic.from.hr> wrote: > > On Mon, Mar 11, 2024 at 11:20:56AM -0400, Alexander Aring wrote: > > when I try doing your steps (btw: thanks for sharing) to run > > fsck.ocfs2 it tells me: > > > > fsck.ocfs2: Unable to access cluster service while initializing the DLM > > > > Looking into strace right before that I see a: > > > > connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = > > -1 ECONNREFUSED (Connection refused) > > > > it's true I don't have ocfs2_controld running, after looking into > > ocfs2-tools source code, I declare it as not compilable with recent > > corosync versions. > > Not sure about that, because I also don't have ocfs2_controld running. > This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack > should be set with: > > echo pcmk > /sys/fs/ocfs2/cluster_stack > I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go. I am able to run fsck and I see on "dlm_tool ls" a lockspace with a similar name you have. The bad news is, I can't reproduce it. After fsck the lockspace releases successfully. I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration? Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM kernel locks. I am asking myself how this is supposed to work. > > btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. > > After I look at your debugging information it would be nice to see if > > the device() read for ast callbacks are there. > > Here is a trace report for the fsck call: > > cpus=1 > fsck.ocfs2-1696 [000] 54.621075: dlm_lock_start: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331 > fsck.ocfs2-1696 [000] 54.692487: dlm_lock_end: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331 > fsck.ocfs2-1696 [000] 54.695447: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331 > fsck.ocfs2-1696 [000] 54.695636: dlm_lock_start: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030 > fsck.ocfs2-1696 [000] 54.766959: dlm_lock_end: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030 > fsck.ocfs2-1696 [000] 54.770765: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030 > fsck.ocfs2-1696 [000] 54.770821: dlm_lock_start: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531 > fsck.ocfs2-1696 [000] 54.845613: dlm_lock_end: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 > fsck.ocfs2-1696 [000] 54.848395: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 > fsck.ocfs2-1696 [000] 54.868951: dlm_unlock_start: ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531 > fsck.ocfs2-1696 [000] 54.958319: dlm_unlock_end: ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 > fsck.ocfs2-1696 [000] 54.982397: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531 > fsck.ocfs2-1696 [000] 54.982450: dlm_lock_start: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531 > fsck.ocfs2-1696 [000] 55.056390: dlm_lock_end: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 > fsck.ocfs2-1696 [000] 55.058946: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 > fsck.ocfs2-1696 [000] 55.082546: dlm_unlock_start: ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531 > fsck.ocfs2-1696 [000] 55.182903: dlm_unlock_end: ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 > fsck.ocfs2-1696 [000] 55.210128: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531 > fsck.ocfs2-1696 [000] 55.530687: dlm_unlock_start: ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030 > fsck.ocfs2-1696 [000] 55.625015: dlm_unlock_end: ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030 > fsck.ocfs2-1696 [000] 55.650054: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030 > fsck.ocfs2-1696 [000] 55.672033: dlm_unlock_start: ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331 > fsck.ocfs2-1696 [000] 55.766844: dlm_unlock_end: ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331 > fsck.ocfs2-1696 [000] 55.791829: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331 > looks okay to me. For dlm user locks there is currently an issue with lvbarea, see [0]. Not sure if this is why you have such issues right now, it depends what fsck is doing with the lvb. - Alex [0] https://lore.kernel.org/gfs2/CAK-6q+h-RngJqJ6Rx2h4a0Qy1j+ZRvCn5E-3Qee7_P3pWXJonQ@mail.gmail.com/T/#ma861c86ab54991a0d9a2742c6983544289f3bdd0 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-11 23:43 ` Alexander Aring @ 2024-03-12 8:09 ` Valentin Vidić 2024-03-12 17:55 ` Alexander Aring 0 siblings, 1 reply; 14+ messages in thread From: Valentin Vidić @ 2024-03-12 8:09 UTC (permalink / raw) To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel [-- Attachment #1: Type: text/plain, Size: 2600 bytes --] On Mon, Mar 11, 2024 at 07:43:41PM -0400, Alexander Aring wrote: > I step forward here, but still I am not able to set a cluster name. I > hardcoded it in the kernel and "it seems" to work. How do you set the > cluster name? It seems that setting the cluster name in mkfs is not > the way to go. I think the cluster name comes from corosync, here is the config I have there: totem { version: 2 cluster_name: debian crypto_cipher: none crypto_hash: none } logging { fileline: off to_stderr: yes to_logfile: yes logfile: /var/log/corosync/corosync.log to_syslog: yes debug: off logger_subsys { subsys: QUORUM debug: off } } quorum { provider: corosync_votequorum } nodelist { node { name: node1 nodeid: 1 ring0_addr: 127.0.0.1 } } The status of the services is as follows: # corosync-quorumtool Quorum information ------------------ Date: Tue Mar 12 07:49:18 2024 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 1 Ring ID: 1.5 Quorate: Yes Votequorum information ---------------------- Expected votes: 1 Highest expected: 1 Total votes: 1 Quorum: 1 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 1 1 node1 (local) # dlm_tool dump_config daemon_debug=0 foreground=1 (cli option) log_debug=0 protocol=detect port=21064 debug_logfile=0 bind_all=0 mark=0 enable_fscontrol=0 enable_plock=1 plock_debug=0 plock_rate_limit=0 plock_ownership=0 drop_resources_time=10000 drop_resources_count=10 drop_resources_age=10000 post_join_delay=30 enable_fencing=1 enable_concurrent_fencing=0 enable_startup_fencing=1 repeat_failed_fencing=1 enable_quorum_fencing=1 enable_quorum_lockspace=1 enable_helper=1 help=-1 version=-1 > I am able to run fsck and I see on "dlm_tool ls" a lockspace with a > similar name you have. > > The bad news is, I can't reproduce it. After fsck the lockspace > releases successfully. > > I see on your debug information that it uses dlm user space locks, in > my case I am using dlm kernel locks and I am asking myself why. Can > you tell me more about your cluster configuration? > > Somehow in your case fsck is using the DLM user space locks > functionality from libdlm (the user space DLM locking library) but in > my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM > kernel locks. > I am asking myself how this is supposed to work. I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs module is not loaded, so it falls back to user space locks? -- Valentin [-- Attachment #2: fsck.1654 --] [-- Type: text/plain, Size: 55814 bytes --] execve("/usr/sbin/fsck.ocfs2", ["fsck.ocfs2", "-f", "-y", "/dev/loop0"], 0x7fff21dd7108 /* 27 vars */) = 0 brk(NULL) = 0x563054c7d000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13065, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 13065, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f61b3401000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b33ff000 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdlm_lt.so.3", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18424, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 20536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f9000 mmap(0x7f61b33fa000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33fa000 mmap(0x7f61b33fc000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fc000 mmap(0x7f61b33fd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fd000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcmap.so.4", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=26672, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 28816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f1000 mmap(0x7f61b33f3000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33f3000 mmap(0x7f61b33f6000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f6000 mmap(0x7f61b33f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f7000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18344, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 20552, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33eb000 mmap(0x7f61b33ed000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33ed000 mmap(0x7f61b33ee000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ee000 mmap(0x7f61b33ef000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ef000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libaio.so.1t64", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14184, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 16432, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33e6000 mmap(0x7f61b33e7000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33e7000 mmap(0x7f61b33e8000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e8000 mmap(0x7f61b33e9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e9000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220x\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1926256, ...}, AT_EMPTY_PATH) = 0 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 1974096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3204000 mmap(0x7f61b322a000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b322a000 mmap(0x7f61b337f000, 344064, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17b000) = 0x7f61b337f000 mmap(0x7f61b33d3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x7f61b33d3000 mmap(0x7f61b33d9000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b33d9000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcorosync_common.so.4", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13952, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 16400, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b31ff000 mmap(0x7f61b3200000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b3200000 mmap(0x7f61b3201000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3201000 mmap(0x7f61b3202000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3202000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libqb.so.100", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=165872, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b31fd000 mmap(NULL, 430816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3193000 mmap(0x7f61b319a000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f61b319a000 mmap(0x7f61b31b1000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) = 0x7f61b31b1000 mmap(0x7f61b31ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x27000) = 0x7f61b31ba000 mmap(0x7f61b31bc000, 262880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b31bc000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libsystemd.so.0", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=918624, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 923208, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30b1000 mmap(0x7f61b30c8000, 573440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f61b30c8000 mmap(0x7f61b3154000, 200704, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa3000) = 0x7f61b3154000 mmap(0x7f61b3185000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd3000) = 0x7f61b3185000 mmap(0x7f61b3192000, 1608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b3192000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200w\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47288, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 45128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30a5000 mmap(0x7f61b30a8000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b30a8000 mmap(0x7f61b30ad000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f61b30ad000 mmap(0x7f61b30af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f61b30af000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=1336728, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 1340360, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f5d000 mmap(0x7f61b2f6c000, 966656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f61b2f6c000 mmap(0x7f61b3058000, 274432, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xfb000) = 0x7f61b3058000 mmap(0x7f61b309b000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13d000) = 0x7f61b309b000 mmap(0x7f61b30a4000, 968, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b30a4000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblz4.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=149952, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 152032, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f37000 mmap(0x7f61b2f3a000, 122880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b2f3a000 mmap(0x7f61b2f58000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f61b2f58000 mmap(0x7f61b2f5b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f61b2f5b000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=247800, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2f35000 mmap(NULL, 245800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2ef8000 mmap(0x7f61b2efc000, 167936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f61b2efc000 mmap(0x7f61b2f25000, 57344, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2d000) = 0x7f61b2f25000 mmap(0x7f61b2f33000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x7f61b2f33000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=784376, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 786496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e37000 mmap(0x7f61b2e3c000, 679936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e3c000 mmap(0x7f61b2ee2000, 81920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xab000) = 0x7f61b2ee2000 mmap(0x7f61b2ef6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xbe000) = 0x7f61b2ef6000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=161864, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 164304, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e0e000 mmap(0x7f61b2e13000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e13000 mmap(0x7f61b2e2a000, 45056, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7f61b2e2a000 mmap(0x7f61b2e35000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b2e35000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e0c000 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e09000 arch_prctl(ARCH_SET_FS, 0x7f61b2e09800) = 0 set_tid_address(0x7f61b2e09ad0) = 1654 set_robust_list(0x7f61b2e09ae0, 24) = 0 rseq(0x7f61b2e0a120, 0x20, 0, 0x53053053) = 0 mprotect(0x7f61b33d3000, 16384, PROT_READ) = 0 mprotect(0x7f61b2e35000, 4096, PROT_READ) = 0 mprotect(0x7f61b2ef6000, 4096, PROT_READ) = 0 mprotect(0x7f61b2f33000, 4096, PROT_READ) = 0 mprotect(0x7f61b2f5b000, 4096, PROT_READ) = 0 mprotect(0x7f61b309b000, 20480, PROT_READ) = 0 mprotect(0x7f61b30af000, 4096, PROT_READ) = 0 mprotect(0x7f61b3185000, 49152, PROT_READ) = 0 mprotect(0x7f61b31ba000, 4096, PROT_READ) = 0 mprotect(0x7f61b3202000, 4096, PROT_READ) = 0 mprotect(0x7f61b33e9000, 4096, PROT_READ) = 0 mprotect(0x7f61b33ef000, 4096, PROT_READ) = 0 mprotect(0x7f61b33f7000, 4096, PROT_READ) = 0 mprotect(0x7f61b33fd000, 4096, PROT_READ) = 0 mprotect(0x563052eed000, 4096, PROT_READ) = 0 mprotect(0x7f61b3435000, 8192, PROT_READ) = 0 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 munmap(0x7f61b3401000, 13065) = 0 getrandom("\xb3\xf1\xb9\xa1\x5c\xd8\xbe\xe2", 8, GRND_NONBLOCK) = 8 brk(NULL) = 0x563054c7d000 brk(0x563054c9e000) = 0x563054c9e000 prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1 prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument) prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1 prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument) prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument) prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument) newfstatat(2, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0 write(2, "fsck.ocfs2 1.8.8\n", 17) = 17 newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0 openat(AT_FDCWD, "/proc/swaps", O_RDONLY) = 3 newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0 read(3, "Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPr"..., 1024) = 39 read(3, "", 1024) = 0 close(3) = 0 openat(AT_FDCWD, "/proc/mounts", O_RDONLY|O_CLOEXEC) = 3 newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0 newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0 read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024 newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0 newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616 newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) read(3, "", 1024) = 0 newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 close(3) = 0 openat(AT_FDCWD, "/etc/mtab", O_RDONLY|O_CLOEXEC) = 3 newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0 newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0 read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024 newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0 newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616 newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory) read(3, "", 1024) = 0 newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 close(3) = 0 newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_EXCL) = 3 close(3) = 0 rt_sigaction(SIGTERM, {sa_handler=0x563052e98e30, sa_mask=[TERM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGINT, {sa_handler=0x563052e98e30, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0 openat(AT_FDCWD, "/dev/loop0", O_RDWR|O_DIRECT) = 3 pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512 uname({sysname="Linux", nodename="debian", ...}) = 0 ioctl(3, BLKROGET, [0]) = 0 pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512 pread64(3, "\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2"..., 512, 1024) = 512 pread64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024 openat(AT_FDCWD, "/sys/fs/ocfs2/cluster_stack", O_RDONLY) = 4 newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=4096, ...}, AT_EMPTY_PATH) = 0 read(4, "pcmk\n", 4096) = 5 close(4) = 0 openat(AT_FDCWD, "/sys/fs/o2cb/interface_revision", O_RDONLY) = 4 read(4, "5\n", 15) = 2 read(4, "", 13) = 0 close(4) = 0 newfstatat(AT_FDCWD, "/sys/kernel/config", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0 statfs("/sys/kernel/config", {f_type=CONFIGFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0x20, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0 rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 4 ioctl(4, BLKSSZGET, [512]) = 0 close(4) = 0 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\6\0\0\0\0\0@\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 14336) = 1024 openat(AT_FDCWD, "/sys/module/ocfs2_dlmfs/parameters/capabilities", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/proc/misc", O_RDONLY) = 4 newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0 read(4, "122 dlm_plock\n123 dlm-monitor\n12"..., 1024) = 220 close(4) = 0 newfstatat(AT_FDCWD, "/dev/misc/dlm-control", {st_mode=S_IFCHR|0666, st_rdev=makedev(0xa, 0x7c), ...}, 0) = 0 openat(AT_FDCWD, "/dev/misc/dlm-control", O_RDWR) = 4 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 read(4, "\6\0\0\0\0\0\0\0\2\0\0\0", 12) = 12 openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5 newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0 read(5, "1\n", 4096) = 2 close(5) = 0 openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 5 read(5, "\221cK\240s\342\265\212", 8) = 8 close(5) = 0 brk(0x563054cc9000) = 0x563054cc9000 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\4\1\0\0\0\0\0\0\0\0\0\00048E8C81F"..., 152) = 121 newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", 0x7ffdc4986d90, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/dev/misc/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5 newfstatat(5, "", {st_mode=S_IFDIR|0755, st_size=120, ...}, AT_EMPTY_PATH) = 0 getdents64(5, 0x563054ca9790 /* 6 entries */, 32768) = 200 newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFCHR|0660, st_rdev=makedev(0xa, 0x79), ...}, 0) = 0 close(5) = 0 openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5 fchmod(5, 0600) = 0 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\3\21\0\0\10\0\0\0\0\0\0\0\0\0\0\0"..., 137) = 1 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0h\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 104 openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 6 newfstatat(6, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0 read(6, "1\n", 4096) = 2 close(6) = 0 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 2 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 3 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\3\0\0\0\0\0\0\0"..., 120) = 0 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 4 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\4\0\0\0\0\0\0\0"..., 120) = 0 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 newfstatat(1, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0 write(1, "Checking OCFS2 filesystem in /de"..., 41) = 41 write(1, " Label: <NONE>\n", 29) = 29 write(1, " UUID: 48E8C81F03"..., 55) = 55 write(1, " Number of blocks: 204800\n", 29) = 29 write(1, " Block size: 1024\n", 27) = 27 write(1, " Number of clusters: 51200\n", 28) = 28 write(1, " Cluster size: 4096\n", 27) = 27 write(1, " Number of slots: 2\n\n", 25) = 25 sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000 munmap(0x7f61afe00000, 50368512) = 0 mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000 mmap(NULL, 3149824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afa00000 sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 mlock(0x7f61afe00400, 50364416) = 0 mlock(0x7f61afa00010, 3147776) = 0 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024 pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 552960) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024 pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 4747264) = 1024 brk(0x563054cea000) = 0x563054cea000 sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=201494528, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 munlock(0x7f61afe00400, 50364416) = 0 munmap(0x7f61afe00000, 50368512) = 0 munlock(0x7f61afa00010, 3147776) = 0 munmap(0x7f61afa00000, 3149824) = 0 mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000 munmap(0x7f61b0600000, 40300544) = 0 mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000 brk(0x563054f61000) = 0x563054f61000 sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=220368896, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 mlock(0x7f61b0600400, 40296448) = 0 mlock(0x563054cd9650, 2518528) = 0 munlock(0x7f61b0600400, 40296448) = 0 munmap(0x7f61b0600000, 40300544) = 0 munlock(0x563054cd9650, 2518528) = 0 mmap(NULL, 20152320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b1a00000 munmap(0x7f61b1a00000, 20152320) = 0 brk(0x563056031000) = 0x563056031000 brk(0x563056165000) = 0x563056165000 sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=216174592, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 mlock(0x563054cd9800, 20148224) = 0 mlock(0x563056010a90, 1259264) = 0 pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 26624) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 27648) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024 pread64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024 write(1, "/dev/loop0 was run with -f, chec"..., 42) = 42 write(1, "Pass 0a: Checking cluster alloca"..., 44) = 44 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33247}, ru_stime={tv_sec=0, tv_usec=546077}, ...}) = 0 pread64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 io_setup(7, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615a800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615ac00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615bc00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615c000, aio_nbytes=1024, aio_offset=188743680}]) = 7 io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x5630561553b0, res=1024, res2=0}, {data=0, obj=0x563056155370, res=1024, res2=0}, {data=0, obj=0x563056155330, res=1024, res2=0}, {data=0, obj=0x5630561552f0, res=1024, res2=0}, {data=0, obj=0x5630561552b0, res=1024, res2=0}, {data=0, obj=0x563056155270, res=1024, res2=0}, {data=0, obj=0x563056155230, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7 io_destroy(0x7f61b3404000) = 0 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33261}, ru_stime={tv_sec=0, tv_usec=546303}, ...}) = 0 write(1, "Pass 0b: Checking inode allocati"..., 42) = 42 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33263}, ru_stime={tv_sec=0, tv_usec=546336}, ...}) = 0 pread64(3, "INODE01\0L\23\t<\377\377\4\0\0\0\0\0D\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 12288) = 1024 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=8192}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 pread64(3, "INODE01\0L\23\t<\377\377\16\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 22528) = 1024 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=13172736}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 brk(0x563056186000) = 0x563056186000 pread64(3, "INODE01\0L\23\t<\377\377\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 23552) = 1024 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33331}, ru_stime={tv_sec=0, tv_usec=547451}, ...}) = 0 write(1, "Pass 0c: Checking extent block a"..., 49) = 49 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33335}, ru_stime={tv_sec=0, tv_usec=547517}, ...}) = 0 pread64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=8941568}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x56305616abb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 pread64(3, "INODE01\0L\23\t<\377\377\r\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 21504) = 1024 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=11038720}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33390}, ru_stime={tv_sec=0, tv_usec=548424}, ...}) = 0 write(1, "Pass 1: Checking inodes and bloc"..., 35) = 35 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33394}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 brk(0x563056597000) = 0x563056597000 pread64(3, "INODE01\0L\23\t<\377\377\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 277504, 9216) = 277504 pread64(3, "INODE01\0L\23\t<\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 13173760) = 1048576 pread64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1047552, 14222336) = 1047552 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38873}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 write(1, "Pass 2: Checking directory entri"..., 35) = 35 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38975}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 io_setup(8, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 8, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176400, aio_nbytes=1024, aio_offset=9216}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176800, aio_nbytes=1024, aio_offset=18432}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176c00, aio_nbytes=1024, aio_offset=19456}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177000, aio_nbytes=1024, aio_offset=286720}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177400, aio_nbytes=1024, aio_offset=287744}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177800, aio_nbytes=1024, aio_offset=288768}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177c00, aio_nbytes=1024, aio_offset=289792}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056178000, aio_nbytes=1024, aio_offset=13173760}]) = 8 io_pgetevents(0x7f61b3404000, 8, 8, [{data=0, obj=0x563056154bb0, res=1024, res2=0}, {data=0, obj=0x563056154b70, res=1024, res2=0}, {data=0, obj=0x563056154b30, res=1024, res2=0}, {data=0, obj=0x563056154af0, res=1024, res2=0}, {data=0, obj=0x563056154ab0, res=1024, res2=0}, {data=0, obj=0x563056154a70, res=1024, res2=0}, {data=0, obj=0x563056154a30, res=1024, res2=0}, {data=0, obj=0x5630561549f0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 8 io_destroy(0x7f61b3404000) = 0 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056163400, aio_nbytes=1024, aio_offset=8192}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056157400, aio_nbytes=1024, aio_offset=13172736}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 write(1, "[DX_TREE_MISSING] Directory 10 i"..., 57) = 57 write(1, " y\n", 3) = 3 pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 pread64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024 pread64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024 io_setup(1, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305657f800, aio_nbytes=1024, aio_offset=8941568}]) = 1 io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1 io_destroy(0x7f61b3404000) = 0 pwrite64(3, "GROUP01\0\0\1\0\10\376\7\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 8941568) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024 pwrite64(3, "DXDIR01\0\211\270\365\260\265\3\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024 pwrite64(3, "DXDIR01\0\237\27d\200\241\1\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pwrite64(3, "DXDIR01\0\262@\254\245\334\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\177\233\217\370\327\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\377\204g\24\336\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0u\236n\322\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\n4z\265\317\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\0b`\367\342\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\303\271\272\342\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\0172\365\275\t\t\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\233\214\21a\213\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\27\363,B\232\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\231\337}\24\31\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\200\264I|\6\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0q\6\316\373U\17\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\261C\321\320k\2\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0!\324',\362\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\343\20\n\340\232\f\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0T\214\2611\3\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\336aa\3229\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0k\274\244\3l\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0s\322D\305\275\32\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\334\304\302\352s\35\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\200+]\37\254\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\241\320\204\4=\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0\25\234Q\5\5\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "DXDIR01\0i\252\342\7\31\34\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024 pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024 pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024 pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024 pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024 pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024 pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024 pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024 pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024 pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024 pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024 pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024 pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=44996}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 write(1, "Pass 3: Checking directory conne"..., 40) = 40 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45065}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45123}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 write(1, "Pass 4a: Checking for orphaned i"..., 38) = 38 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45237}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45351}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 write(1, "Pass 4b: Checking inodes link co"..., 37) = 37 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45395}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45426}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 write(1, "Pass 5: Checking quota informati"..., 35) = 35 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45474}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 brk(0x5630569a1000) = 0x5630569a1000 io_setup(7, [0x7f61b3404000]) = 0 io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196c00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197c00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056198000, aio_nbytes=1024, aio_offset=188743680}]) = 7 io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x56305616ffb0, res=1024, res2=0}, {data=0, obj=0x56305616ff70, res=1024, res2=0}, {data=0, obj=0x56305616ff30, res=1024, res2=0}, {data=0, obj=0x56305616fef0, res=1024, res2=0}, {data=0, obj=0x56305616feb0, res=1024, res2=0}, {data=0, obj=0x56305616fe70, res=1024, res2=0}, {data=0, obj=0x56305616fe30, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7 io_destroy(0x7f61b3404000) = 0 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024 pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13139968) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024 pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024 pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024 pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024 pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024 pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024 pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13144064) = 2048 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024 pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13148160) = 2048 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024 pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13152256) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024 pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024 pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024 pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024 pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024 pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024 pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13156352) = 2048 pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024 pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024 pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13160448) = 2048 getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=55343}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0 write(1, "All passes succeeded.\n", 22) = 22 write(1, "\n", 1) = 1 pwrite64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024 pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 13135872) = 4096 rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\355\376\20\0\0\0\10\0\0\0\2\0\0\0\377\377\377\377"..., 120) = 0 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0 \0\230\304\10\0\0\0\1\0\0\0\310\16\355\376"..., 120) = 0 read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFLNK|0777, st_size=39, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(5, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0xa, 0x79), ...}, AT_EMPTY_PATH) = 0 close(5) = 0 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\5\1\0\0\0\0\0\0y\0\0\0\4\0\0\0\0\0\0\0"..., 120) = -1 EBUSY (Device or resource busy) openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5 close(5) = 0 openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 5 ioctl(5, BLKSSZGET, [512]) = 0 close(5) = 0 openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5 newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0 read(5, "1\n", 4096) = 2 close(5) = 0 rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024 pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024 pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024 munlock(0x563054cd9800, 20148224) = 0 munlock(0x563056010a90, 1259264) = 0 close(3) = 0 exit_group(0) = ? +++ exited with 0 +++ ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-12 8:09 ` Valentin Vidić @ 2024-03-12 17:55 ` Alexander Aring 2024-03-12 18:37 ` Valentin Vidić 0 siblings, 1 reply; 14+ messages in thread From: Alexander Aring @ 2024-03-12 17:55 UTC (permalink / raw) To: Valentin Vidić Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel Hi, On Tue, Mar 12, 2024 at 4:09 AM Valentin Vidić <vvidic@valentin-vidic.from.hr> wrote: > > On Mon, Mar 11, 2024 at 07:43:41PM -0400, Alexander Aring wrote: > > I step forward here, but still I am not able to set a cluster name. I > > hardcoded it in the kernel and "it seems" to work. How do you set the > > cluster name? It seems that setting the cluster name in mkfs is not > > the way to go. > > I think the cluster name comes from corosync, here is the config I have > there: > > totem { > version: 2 > cluster_name: debian > crypto_cipher: none > crypto_hash: none > } > logging { > fileline: off > to_stderr: yes > to_logfile: yes > logfile: /var/log/corosync/corosync.log > to_syslog: yes > debug: off > logger_subsys { > subsys: QUORUM > debug: off > } > } > quorum { > provider: corosync_votequorum > } > nodelist { > node { > name: node1 > nodeid: 1 > ring0_addr: 127.0.0.1 > } > } > > The status of the services is as follows: > > # corosync-quorumtool > Quorum information > ------------------ > Date: Tue Mar 12 07:49:18 2024 > Quorum provider: corosync_votequorum > Nodes: 1 > Node ID: 1 > Ring ID: 1.5 > Quorate: Yes > > Votequorum information > ---------------------- > Expected votes: 1 > Highest expected: 1 > Total votes: 1 > Quorum: 1 > Flags: Quorate > > Membership information > ---------------------- > Nodeid Votes Name > 1 1 node1 (local) > > # dlm_tool dump_config > daemon_debug=0 > foreground=1 (cli option) > log_debug=0 > protocol=detect > port=21064 > debug_logfile=0 > bind_all=0 > mark=0 > enable_fscontrol=0 > enable_plock=1 > plock_debug=0 > plock_rate_limit=0 > plock_ownership=0 > drop_resources_time=10000 > drop_resources_count=10 > drop_resources_age=10000 > post_join_delay=30 > enable_fencing=1 > enable_concurrent_fencing=0 > enable_startup_fencing=1 > repeat_failed_fencing=1 > enable_quorum_fencing=1 > enable_quorum_lockspace=1 > enable_helper=1 > help=-1 > version=-1 > > > I am able to run fsck and I see on "dlm_tool ls" a lockspace with a > > similar name you have. > > > > The bad news is, I can't reproduce it. After fsck the lockspace > > releases successfully. > > > > I see on your debug information that it uses dlm user space locks, in > > my case I am using dlm kernel locks and I am asking myself why. Can > > you tell me more about your cluster configuration? > > > > Somehow in your case fsck is using the DLM user space locks > > functionality from libdlm (the user space DLM locking library) but in > > my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM > > kernel locks. > > I am asking myself how this is supposed to work. > > I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs > module is not loaded, so it falls back to user space locks? I was able to reproduce the issue and sent a fix out. If you want to extend your debian testing using different approach (not libdlm) this can be done by: mount -t ocfs2_dlmfs none /dlm this will somehow do some UAPI that triggers dlm kernel locking (so far I see it). How it is intended to work and which combination works or not? I have no idea, all I can say is that you should avoid using the same lockspace "name" for libdlm user and kernel dlm locking. I need to recheck if we have a check for that, that might break stuff but should not be used in such a way anyway... and maybe in future there is no difference between kernel/user locks. I am curious about your testing at debian cluster, do you also test gfs2? - Alex ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-12 17:55 ` Alexander Aring @ 2024-03-12 18:37 ` Valentin Vidić 2024-03-12 19:55 ` Alexander Aring 0 siblings, 1 reply; 14+ messages in thread From: Valentin Vidić @ 2024-03-12 18:37 UTC (permalink / raw) To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel On Tue, Mar 12, 2024 at 01:55:54PM -0400, Alexander Aring wrote: > I was able to reproduce the issue and sent a fix out. Yay, thanks a lot for testing, I can confirm the patches fix the problem for me! > If you want to extend your debian testing using different approach > (not libdlm) this can be done by: > > mount -t ocfs2_dlmfs none /dlm I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs is only relevant when used with o2cb cluster type instead of corosync. > I am curious about your testing at debian cluster, do you also test gfs2? Yes, a similar test for gfs2 would be this: https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads It is not really a cluster test since it only runs on one KVM machine. But as a package test it works well because even for 1-node cluster all the code (userspace and kernel) still needs to work correctly. -- Valentin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-12 18:37 ` Valentin Vidić @ 2024-03-12 19:55 ` Alexander Aring 2024-03-12 20:03 ` Alexander Aring 2024-03-12 20:42 ` Valentin Vidić 0 siblings, 2 replies; 14+ messages in thread From: Alexander Aring @ 2024-03-12 19:55 UTC (permalink / raw) To: Valentin Vidić Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel Hi, On Tue, Mar 12, 2024 at 2:37 PM Valentin Vidić <vvidic@valentin-vidic.from.hr> wrote: > > On Tue, Mar 12, 2024 at 01:55:54PM -0400, Alexander Aring wrote: > > I was able to reproduce the issue and sent a fix out. > > Yay, thanks a lot for testing, I can confirm the patches fix the problem > for me! > Then you can reply to the mail with: Tested-by: Valentin Vidić <vvidic@valentin-vidic.from.hr> that would help to bring the fix upstream. > > If you want to extend your debian testing using different approach > > (not libdlm) this can be done by: > > > > mount -t ocfs2_dlmfs none /dlm > > I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs > is only relevant when used with o2cb cluster type instead of corosync. > Yes, I think you are right. It is a replacement for corosync/dlm_controld handling. It seems I had both setup working at the same time, but it still uses kernel DLM locking. > > I am curious about your testing at debian cluster, do you also test gfs2? > > Yes, a similar test for gfs2 would be this: > > https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads > > It is not really a cluster test since it only runs on one KVM > machine. But as a package test it works well because even for > 1-node cluster all the code (userspace and kernel) still needs > to work correctly. > I see, I currently have plans/ideas to add namespace support into kernel DLM that might help you by extending your testing. The general idea would be that you can do something like: ... ip netns exec node1 mount /dev/fake_shared_block /mnt/node1 ip netns exec node2 mount /dev/fake_shared_block /mnt/node2 ip netns exec node3 mount /dev/fake_shared_block /mnt/node3 ... I hope that somehow explains the idea behind it. Setup your cluster networking with namespaces, e.g. selftests of Linux kernel is doing it [0] and the DLM lockspaces are separated by net namespaces*. It "should" run like a cluster with several machines, it is only to do more testing and we don't need to synchronize tests over the network. However it will _not_ replace real cluster testing but it is a starting point to make testing "simpler". Somewhere is even a big fat "software" switch you can run wireshark on it to see all cluster traffic going on. * In theory I am only abusing net namespaces for separate DLM lockspaces, but I don't want to introduce another kind of namespace now. However net namespaces fit here because it is necessary for a per node networking setup anyway. - Alex [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/net/icmp.sh ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-12 19:55 ` Alexander Aring @ 2024-03-12 20:03 ` Alexander Aring 2024-03-12 20:42 ` Valentin Vidić 1 sibling, 0 replies; 14+ messages in thread From: Alexander Aring @ 2024-03-12 20:03 UTC (permalink / raw) To: Valentin Vidić Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel Hi, On Tue, Mar 12, 2024 at 3:55 PM Alexander Aring <aahringo@redhat.com> wrote: > > Hi, > > On Tue, Mar 12, 2024 at 2:37 PM Valentin Vidić > <vvidic@valentin-vidic.from.hr> wrote: > > > > On Tue, Mar 12, 2024 at 01:55:54PM -0400, Alexander Aring wrote: > > > I was able to reproduce the issue and sent a fix out. > > > > Yay, thanks a lot for testing, I can confirm the patches fix the problem > > for me! > > > > Then you can reply to the mail with: > > Tested-by: Valentin Vidić <vvidic@valentin-vidic.from.hr> > > that would help to bring the fix upstream. > > > > If you want to extend your debian testing using different approach > > > (not libdlm) this can be done by: > > > > > > mount -t ocfs2_dlmfs none /dlm > > > > I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs > > is only relevant when used with o2cb cluster type instead of corosync. > > > > Yes, I think you are right. It is a replacement for > corosync/dlm_controld handling. It seems I had both setup working at > the same time, but it still uses kernel DLM locking. It also requires a ocfs2-tools tool at a specific location: /sbin/ocfs2_hb_ctl maybe this is your issue why it wasn't working. My kernel was complaining about it because it was missing in /sbin/ - Alex ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error 2024-03-12 19:55 ` Alexander Aring 2024-03-12 20:03 ` Alexander Aring @ 2024-03-12 20:42 ` Valentin Vidić 1 sibling, 0 replies; 14+ messages in thread From: Valentin Vidić @ 2024-03-12 20:42 UTC (permalink / raw) To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel On Tue, Mar 12, 2024 at 03:55:46PM -0400, Alexander Aring wrote: > Then you can reply to the mail with: > > Tested-by: Valentin Vidić <vvidic@valentin-vidic.from.hr> > > that would help to bring the fix upstream. Sure, replied to the other thread. > I see, I currently have plans/ideas to add namespace support into > kernel DLM that might help you by extending your testing. The general > idea would be that you can do something like: > > ... > ip netns exec node1 mount /dev/fake_shared_block /mnt/node1 > ip netns exec node2 mount /dev/fake_shared_block /mnt/node2 > ip netns exec node3 mount /dev/fake_shared_block /mnt/node3 > ... Yeah, that would definitely be cool. And thanks for the pointers, I will definitely try to use namespaces for some simpler corosync tests first. > It also requires a ocfs2-tools tool at a specific location: > > /sbin/ocfs2_hb_ctl > > maybe this is your issue why it wasn't working. My kernel was > complaining about it because it was missing in /sbin/ That binary is already in the Debian package. I think the error fsck reported was something like "Could not create domain". -- Valentin ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2024-03-12 20:42 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-03-10 21:46 ocfs2 mount error Valentin Vidić 2024-03-11 8:37 ` Joseph Qi 2024-03-11 9:02 ` Valentin Vidić 2024-03-11 9:27 ` Joseph Qi 2024-03-11 12:28 ` Heming Zhao 2024-03-11 15:20 ` Alexander Aring 2024-03-11 19:27 ` Valentin Vidić 2024-03-11 23:43 ` Alexander Aring 2024-03-12 8:09 ` Valentin Vidić 2024-03-12 17:55 ` Alexander Aring 2024-03-12 18:37 ` Valentin Vidić 2024-03-12 19:55 ` Alexander Aring 2024-03-12 20:03 ` Alexander Aring 2024-03-12 20:42 ` Valentin Vidić
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox