* ocfs2 mount error
@ 2024-03-10 21:46 Valentin Vidić
2024-03-11 8:37 ` Joseph Qi
0 siblings, 1 reply; 14+ messages in thread
From: Valentin Vidić @ 2024-03-10 21:46 UTC (permalink / raw)
To: ocfs2-devel, Alexander Aring, David Teigland, gfs2
Hi,
I've started seeing the following mount error on recent kernels (6.7, 6.8)
while the same script worked on older versions (6.1):
2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user
2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace
2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17
2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17
2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17
Script runs the commands in the following order:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2
fsck command tries to release the lockspace but it seems to be busy so
it stays around and the mount command fails. Running dlm_tool leave
manually removes it and after that the mount command works again.
A git bisect has led me to the following change:
61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks
Could it be that some callback does not run correctly causing the lockspace
to be busy? I have added some more debug info below including some logging of
5 kref_get and 4 kref_put that could be related.
# dlm_tool ls
dlm lockspaces
name D6DCCE389FC84C8FB9E99ABE1D941825
id 0x3a105115
flags 0x00000000
change member 1 joined 1 remove 0 failed 0 seq 1,1
members 1
# dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825
cluster nodeid 1 quorate 1 ring seq 5 5
daemon now 1831 fence_pid 0
node 1 M add 54 rem 0 fail 0 fence 0 at 0 0
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000"
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21"
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda"
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda"
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags
2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1
2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,436839+00:00 Call Trace:
2024-03-10T21:04:29,437077+00:00 <TASK>
2024-03-10T21:04:29,437308+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,437653+00:00 add_lkb+0x2f/0x300 [dlm]
2024-03-10T21:04:29,438000+00:00 ? rsb_insert+0x85/0xd0 [dlm]
2024-03-10T21:04:29,438378+00:00 _grant_lock+0x111/0x170 [dlm]
2024-03-10T21:04:29,438756+00:00 do_request+0x71/0x90 [dlm]
2024-03-10T21:04:29,439112+00:00 _request_lock+0x19b/0x270 [dlm]
2024-03-10T21:04:29,439509+00:00 request_lock+0xc7/0x110 [dlm]
2024-03-10T21:04:29,439887+00:00 dlm_user_request+0x10d/0x370 [dlm]
2024-03-10T21:04:29,440304+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,440763+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,441161+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,441578+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,442016+00:00 device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,442413+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:29,442723+00:00 ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,443082+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:29,443389+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,443729+00:00 ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,444086+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1
2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,455462+00:00 Call Trace:
2024-03-10T21:04:29,455922+00:00 <TASK>
2024-03-10T21:04:29,456345+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,456903+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm]
2024-03-10T21:04:29,457548+00:00 do_request+0x7b/0x90 [dlm]
2024-03-10T21:04:29,458128+00:00 _request_lock+0x19b/0x270 [dlm]
2024-03-10T21:04:29,458771+00:00 request_lock+0xc7/0x110 [dlm]
2024-03-10T21:04:29,459375+00:00 dlm_user_request+0x10d/0x370 [dlm]
2024-03-10T21:04:29,460021+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,460714+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,461363+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,462003+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,462659+00:00 device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,463268+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:29,463791+00:00 ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,464359+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:29,464891+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,465440+00:00 ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,466005+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1
2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,477545+00:00 Call Trace:
2024-03-10T21:04:29,478031+00:00 <TASK>
2024-03-10T21:04:29,478494+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,479094+00:00 dlm_user_request+0x299/0x370 [dlm]
2024-03-10T21:04:29,479787+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,480544+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,481195+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,481848+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,482518+00:00 device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,483140+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:29,483672+00:00 ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,484254+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:29,484785+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,485380+00:00 ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,485954+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1
2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,419629+00:00 Call Trace:
2024-03-10T21:04:30,420115+00:00 <TASK>
2024-03-10T21:04:30,420566+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,421157+00:00 find_lkb+0x4e/0xd0 [dlm]
2024-03-10T21:04:30,421758+00:00 dlm_user_unlock+0x6b/0x2f0 [dlm]
2024-03-10T21:04:30,422422+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,423140+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,423785+00:00 device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,424404+00:00 ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,425057+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:30,425606+00:00 ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,426268+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,426857+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:30,427415+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,427969+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,428534+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,429098+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,429657+00:00 ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,430211+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1
2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,443065+00:00 Call Trace:
2024-03-10T21:04:30,443571+00:00 <TASK>
2024-03-10T21:04:30,444029+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,444618+00:00 _unlock_lock+0x73/0x110 [dlm]
2024-03-10T21:04:30,445257+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,445979+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,446641+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,447345+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,447979+00:00 device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,448593+00:00 ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,449460+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:30,450394+00:00 ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,451160+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,451773+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:30,452316+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,452894+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,453491+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,454053+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,454621+00:00 ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,455175+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1
2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,466737+00:00 Call Trace:
2024-03-10T21:04:30,467220+00:00 <TASK>
2024-03-10T21:04:30,467684+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,468276+00:00 _unlock_lock+0x9e/0x110 [dlm]
2024-03-10T21:04:30,468912+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,469607+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,470261+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,470961+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,471611+00:00 device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,472221+00:00 ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,472867+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:30,473397+00:00 ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,474055+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,474633+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:30,475179+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,475759+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,476322+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,476883+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,477442+00:00 ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,478000+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1
2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,489384+00:00 Call Trace:
2024-03-10T21:04:30,489869+00:00 <TASK>
2024-03-10T21:04:30,490320+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,490918+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm]
2024-03-10T21:04:30,491609+00:00 _unlock_lock+0xbe/0x110 [dlm]
2024-03-10T21:04:30,492231+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,492914+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,493553+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,494238+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,494869+00:00 device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,495479+00:00 ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,496133+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:30,496660+00:00 ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,497318+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,497898+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:30,498428+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,498978+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,499532+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,500086+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,500639+00:00 ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,501192+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1
2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,512564+00:00 Call Trace:
2024-03-10T21:04:30,513045+00:00 <TASK>
2024-03-10T21:04:30,513496+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,514081+00:00 __put_lkb+0x30/0x100 [dlm]
2024-03-10T21:04:30,514683+00:00 dlm_user_add_ast+0x1cc/0x300 [dlm]
2024-03-10T21:04:30,515337+00:00 _unlock_lock+0xbe/0x110 [dlm]
2024-03-10T21:04:30,515953+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,516633+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,517265+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,517957+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,518586+00:00 device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,519199+00:00 ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,519840+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:30,520365+00:00 ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,521021+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,521586+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:30,522100+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,522654+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,523212+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,523779+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,524333+00:00 ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,524888+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1
2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,536247+00:00 Call Trace:
2024-03-10T21:04:30,536729+00:00 <TASK>
2024-03-10T21:04:30,537179+00:00 dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,537755+00:00 __put_lkb+0x30/0x100 [dlm]
2024-03-10T21:04:30,538355+00:00 dlm_user_unlock+0x213/0x2f0 [dlm]
2024-03-10T21:04:30,539003+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,539698+00:00 ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,540319+00:00 device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,540927+00:00 ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,541569+00:00 vfs_write+0xec/0x470
2024-03-10T21:04:30,542093+00:00 ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,542759+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,543327+00:00 ksys_write+0x6b/0xf0
2024-03-10T21:04:30,543851+00:00 do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,544395+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,544949+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,545500+00:00 ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,546059+00:00 ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,546621+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0
--
Valentin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-10 21:46 ocfs2 mount error Valentin Vidić
@ 2024-03-11 8:37 ` Joseph Qi
2024-03-11 9:02 ` Valentin Vidić
0 siblings, 1 reply; 14+ messages in thread
From: Joseph Qi @ 2024-03-11 8:37 UTC (permalink / raw)
To: Valentin Vidić, ocfs2-devel, Alexander Aring, David Teigland,
gfs2, ocfs2-devel
Hi,
It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved.
Could you please give more detail on how to reproduce it?
Thanks,
Joseph
On 3/11/24 5:46 AM, Valentin Vidić wrote:
> Hi,
>
> I've started seeing the following mount error on recent kernels (6.7, 6.8)
> while the same script worked on older versions (6.1):
>
> 2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user
> 2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace
> 2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17
> 2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17
> 2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17
>
> Script runs the commands in the following order:
>
> mkfs.ocfs2
> fsck.ocfs2
> mount.ocfs2
>
> fsck command tries to release the lockspace but it seems to be busy so
> it stays around and the mount command fails. Running dlm_tool leave
> manually removes it and after that the mount command works again.
>
> A git bisect has led me to the following change:
>
> 61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks
>
> Could it be that some callback does not run correctly causing the lockspace
> to be busy? I have added some more debug info below including some logging of
> 5 kref_get and 4 kref_put that could be related.
>
> # dlm_tool ls
> dlm lockspaces
> name D6DCCE389FC84C8FB9E99ABE1D941825
> id 0x3a105115
> flags 0x00000000
> change member 1 joined 1 remove 0 failed 0 seq 1,1
> members 1
>
> # dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825
> cluster nodeid 1 quorate 1 ring seq 5 5
> daemon now 1831 fence_pid 0
> node 1 M add 54 rem 0 fail 0 fence 0 at 0 0
>
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000"
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21"
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda"
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda"
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name
> /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags
>
> 2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1
> 2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:29,436839+00:00 Call Trace:
> 2024-03-10T21:04:29,437077+00:00 <TASK>
> 2024-03-10T21:04:29,437308+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:29,437653+00:00 add_lkb+0x2f/0x300 [dlm]
> 2024-03-10T21:04:29,438000+00:00 ? rsb_insert+0x85/0xd0 [dlm]
> 2024-03-10T21:04:29,438378+00:00 _grant_lock+0x111/0x170 [dlm]
> 2024-03-10T21:04:29,438756+00:00 do_request+0x71/0x90 [dlm]
> 2024-03-10T21:04:29,439112+00:00 _request_lock+0x19b/0x270 [dlm]
> 2024-03-10T21:04:29,439509+00:00 request_lock+0xc7/0x110 [dlm]
> 2024-03-10T21:04:29,439887+00:00 dlm_user_request+0x10d/0x370 [dlm]
> 2024-03-10T21:04:29,440304+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:29,440763+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:29,441161+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm]
> 2024-03-10T21:04:29,441578+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm]
> 2024-03-10T21:04:29,442016+00:00 device_write+0x5f1/0x790 [dlm]
> 2024-03-10T21:04:29,442413+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:29,442723+00:00 ? __x64_sys_fcntl+0x90/0xc0
> 2024-03-10T21:04:29,443082+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:29,443389+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:29,443729+00:00 ? exc_page_fault+0x73/0x170
> 2024-03-10T21:04:29,444086+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1
> 2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:29,455462+00:00 Call Trace:
> 2024-03-10T21:04:29,455922+00:00 <TASK>
> 2024-03-10T21:04:29,456345+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:29,456903+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm]
> 2024-03-10T21:04:29,457548+00:00 do_request+0x7b/0x90 [dlm]
> 2024-03-10T21:04:29,458128+00:00 _request_lock+0x19b/0x270 [dlm]
> 2024-03-10T21:04:29,458771+00:00 request_lock+0xc7/0x110 [dlm]
> 2024-03-10T21:04:29,459375+00:00 dlm_user_request+0x10d/0x370 [dlm]
> 2024-03-10T21:04:29,460021+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:29,460714+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:29,461363+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm]
> 2024-03-10T21:04:29,462003+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm]
> 2024-03-10T21:04:29,462659+00:00 device_write+0x5f1/0x790 [dlm]
> 2024-03-10T21:04:29,463268+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:29,463791+00:00 ? __x64_sys_fcntl+0x90/0xc0
> 2024-03-10T21:04:29,464359+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:29,464891+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:29,465440+00:00 ? exc_page_fault+0x73/0x170
> 2024-03-10T21:04:29,466005+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1
> 2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:29,477545+00:00 Call Trace:
> 2024-03-10T21:04:29,478031+00:00 <TASK>
> 2024-03-10T21:04:29,478494+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:29,479094+00:00 dlm_user_request+0x299/0x370 [dlm]
> 2024-03-10T21:04:29,479787+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:29,480544+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:29,481195+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm]
> 2024-03-10T21:04:29,481848+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm]
> 2024-03-10T21:04:29,482518+00:00 device_write+0x5f1/0x790 [dlm]
> 2024-03-10T21:04:29,483140+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:29,483672+00:00 ? __x64_sys_fcntl+0x90/0xc0
> 2024-03-10T21:04:29,484254+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:29,484785+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:29,485380+00:00 ? exc_page_fault+0x73/0x170
> 2024-03-10T21:04:29,485954+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1
> 2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:30,419629+00:00 Call Trace:
> 2024-03-10T21:04:30,420115+00:00 <TASK>
> 2024-03-10T21:04:30,420566+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:30,421157+00:00 find_lkb+0x4e/0xd0 [dlm]
> 2024-03-10T21:04:30,421758+00:00 dlm_user_unlock+0x6b/0x2f0 [dlm]
> 2024-03-10T21:04:30,422422+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:30,423140+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:30,423785+00:00 device_write+0x5bd/0x790 [dlm]
> 2024-03-10T21:04:30,424404+00:00 ? security_file_permission+0x32/0x50
> 2024-03-10T21:04:30,425057+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:30,425606+00:00 ? syscall_exit_to_user_mode+0x81/0x220
> 2024-03-10T21:04:30,426268+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,426857+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:30,427415+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:30,427969+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,428534+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,429098+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,429657+00:00 ? __irq_exit_rcu+0x3b/0xb0
> 2024-03-10T21:04:30,430211+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1
> 2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:30,443065+00:00 Call Trace:
> 2024-03-10T21:04:30,443571+00:00 <TASK>
> 2024-03-10T21:04:30,444029+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:30,444618+00:00 _unlock_lock+0x73/0x110 [dlm]
> 2024-03-10T21:04:30,445257+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
> 2024-03-10T21:04:30,445979+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
> 2024-03-10T21:04:30,446641+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:30,447345+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:30,447979+00:00 device_write+0x5bd/0x790 [dlm]
> 2024-03-10T21:04:30,448593+00:00 ? security_file_permission+0x32/0x50
> 2024-03-10T21:04:30,449460+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:30,450394+00:00 ? syscall_exit_to_user_mode+0x81/0x220
> 2024-03-10T21:04:30,451160+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,451773+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:30,452316+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:30,452894+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,453491+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,454053+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,454621+00:00 ? __irq_exit_rcu+0x3b/0xb0
> 2024-03-10T21:04:30,455175+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1
> 2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:30,466737+00:00 Call Trace:
> 2024-03-10T21:04:30,467220+00:00 <TASK>
> 2024-03-10T21:04:30,467684+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:30,468276+00:00 _unlock_lock+0x9e/0x110 [dlm]
> 2024-03-10T21:04:30,468912+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
> 2024-03-10T21:04:30,469607+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
> 2024-03-10T21:04:30,470261+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:30,470961+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:30,471611+00:00 device_write+0x5bd/0x790 [dlm]
> 2024-03-10T21:04:30,472221+00:00 ? security_file_permission+0x32/0x50
> 2024-03-10T21:04:30,472867+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:30,473397+00:00 ? syscall_exit_to_user_mode+0x81/0x220
> 2024-03-10T21:04:30,474055+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,474633+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:30,475179+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:30,475759+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,476322+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,476883+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,477442+00:00 ? __irq_exit_rcu+0x3b/0xb0
> 2024-03-10T21:04:30,478000+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1
> 2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:30,489384+00:00 Call Trace:
> 2024-03-10T21:04:30,489869+00:00 <TASK>
> 2024-03-10T21:04:30,490320+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:30,490918+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm]
> 2024-03-10T21:04:30,491609+00:00 _unlock_lock+0xbe/0x110 [dlm]
> 2024-03-10T21:04:30,492231+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
> 2024-03-10T21:04:30,492914+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
> 2024-03-10T21:04:30,493553+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:30,494238+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:30,494869+00:00 device_write+0x5bd/0x790 [dlm]
> 2024-03-10T21:04:30,495479+00:00 ? security_file_permission+0x32/0x50
> 2024-03-10T21:04:30,496133+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:30,496660+00:00 ? syscall_exit_to_user_mode+0x81/0x220
> 2024-03-10T21:04:30,497318+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,497898+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:30,498428+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:30,498978+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,499532+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,500086+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,500639+00:00 ? __irq_exit_rcu+0x3b/0xb0
> 2024-03-10T21:04:30,501192+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1
> 2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:30,512564+00:00 Call Trace:
> 2024-03-10T21:04:30,513045+00:00 <TASK>
> 2024-03-10T21:04:30,513496+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:30,514081+00:00 __put_lkb+0x30/0x100 [dlm]
> 2024-03-10T21:04:30,514683+00:00 dlm_user_add_ast+0x1cc/0x300 [dlm]
> 2024-03-10T21:04:30,515337+00:00 _unlock_lock+0xbe/0x110 [dlm]
> 2024-03-10T21:04:30,515953+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm]
> 2024-03-10T21:04:30,516633+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm]
> 2024-03-10T21:04:30,517265+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:30,517957+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:30,518586+00:00 device_write+0x5bd/0x790 [dlm]
> 2024-03-10T21:04:30,519199+00:00 ? security_file_permission+0x32/0x50
> 2024-03-10T21:04:30,519840+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:30,520365+00:00 ? syscall_exit_to_user_mode+0x81/0x220
> 2024-03-10T21:04:30,521021+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,521586+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:30,522100+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:30,522654+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,523212+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,523779+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,524333+00:00 ? __irq_exit_rcu+0x3b/0xb0
> 2024-03-10T21:04:30,524888+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
>
> 2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1
> 2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
> 2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> 2024-03-10T21:04:30,536247+00:00 Call Trace:
> 2024-03-10T21:04:30,536729+00:00 <TASK>
> 2024-03-10T21:04:30,537179+00:00 dump_stack_lvl+0x43/0x60
> 2024-03-10T21:04:30,537755+00:00 __put_lkb+0x30/0x100 [dlm]
> 2024-03-10T21:04:30,538355+00:00 dlm_user_unlock+0x213/0x2f0 [dlm]
> 2024-03-10T21:04:30,539003+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420
> 2024-03-10T21:04:30,539698+00:00 ? device_write+0x44/0x790 [dlm]
> 2024-03-10T21:04:30,540319+00:00 device_write+0x5bd/0x790 [dlm]
> 2024-03-10T21:04:30,540927+00:00 ? security_file_permission+0x32/0x50
> 2024-03-10T21:04:30,541569+00:00 vfs_write+0xec/0x470
> 2024-03-10T21:04:30,542093+00:00 ? syscall_exit_to_user_mode+0x81/0x220
> 2024-03-10T21:04:30,542759+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,543327+00:00 ksys_write+0x6b/0xf0
> 2024-03-10T21:04:30,543851+00:00 do_syscall_64+0x87/0x1a0
> 2024-03-10T21:04:30,544395+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,544949+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,545500+00:00 ? do_syscall_64+0x93/0x1a0
> 2024-03-10T21:04:30,546059+00:00 ? __irq_exit_rcu+0x3b/0xb0
> 2024-03-10T21:04:30,546621+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
> 2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 8:37 ` Joseph Qi
@ 2024-03-11 9:02 ` Valentin Vidić
2024-03-11 9:27 ` Joseph Qi
2024-03-11 15:20 ` Alexander Aring
0 siblings, 2 replies; 14+ messages in thread
From: Valentin Vidić @ 2024-03-11 9:02 UTC (permalink / raw)
To: Joseph Qi; +Cc: ocfs2-devel, Alexander Aring, David Teigland, gfs2, ocfs2-devel
On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote:
> It seems that you are using fsdlm.
> But from your running commands, I don't see how fsdlm get involved.
> Could you please give more detail on how to reproduce it?
Sure, this is the original Pacemaker test script that failed:
https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
Basically it starts Corosync and DLM services and than calls:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2
--
Valentin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 9:02 ` Valentin Vidić
@ 2024-03-11 9:27 ` Joseph Qi
2024-03-11 12:28 ` Heming Zhao
2024-03-11 15:20 ` Alexander Aring
1 sibling, 1 reply; 14+ messages in thread
From: Joseph Qi @ 2024-03-11 9:27 UTC (permalink / raw)
To: Valentin Vidić
Cc: Alexander Aring, David Teigland, gfs2, ocfs2-devel, Heming Zhao
On 3/11/24 5:02 PM, Valentin Vidić wrote:
> On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote:
>> It seems that you are using fsdlm.
>> But from your running commands, I don't see how fsdlm get involved.
>> Could you please give more detail on how to reproduce it?
>
> Sure, this is the original Pacemaker test script that failed:
>
> https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
>
> Basically it starts Corosync and DLM services and than calls:
>
> mkfs.ocfs2
> fsck.ocfs2
> mount.ocfs2
>
Thanks for quick reply.
Now it confirms that it is using fsdlm.
Joseph
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 9:27 ` Joseph Qi
@ 2024-03-11 12:28 ` Heming Zhao
0 siblings, 0 replies; 14+ messages in thread
From: Heming Zhao @ 2024-03-11 12:28 UTC (permalink / raw)
To: Joseph Qi, Valentin Vidić
Cc: Alexander Aring, David Teigland, gfs2, ocfs2-devel
On 3/11/24 17:27, Joseph Qi wrote:
>
>
> On 3/11/24 5:02 PM, Valentin Vidić wrote:
>> On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote:
>>> It seems that you are using fsdlm.
>>> But from your running commands, I don't see how fsdlm get involved.
>>> Could you please give more detail on how to reproduce it?
>>
>> Sure, this is the original Pacemaker test script that failed:
>>
>> https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
>>
>> Basically it starts Corosync and DLM services and than calls:
>>
>> mkfs.ocfs2
>> fsck.ocfs2
>> mount.ocfs2
>>
>
> Thanks for quick reply.
> Now it confirms that it is using fsdlm.
>
> Joseph
>
>
It looks like I met the same issue one year ago.
check [1] and search for "error 17." in this mail.
[1]: https://lore.kernel.org/ocfs2-devel/20230626150916.je3egonzb3crvbl5@p15/
If my memory is still right, this root cause of this issue is that
fsck forgets to clear the lockspace before exiting. I will investigate
the debian's issue and provide a fix.
Thanks,
Heming
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 9:02 ` Valentin Vidić
2024-03-11 9:27 ` Joseph Qi
@ 2024-03-11 15:20 ` Alexander Aring
2024-03-11 19:27 ` Valentin Vidić
1 sibling, 1 reply; 14+ messages in thread
From: Alexander Aring @ 2024-03-11 15:20 UTC (permalink / raw)
To: Valentin Vidić
Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
Hi,
On Mon, Mar 11, 2024 at 5:02 AM Valentin Vidić
<vvidic@valentin-vidic.from.hr> wrote:
>
> On Mon, Mar 11, 2024 at 04:37:39PM +0800, Joseph Qi wrote:
> > It seems that you are using fsdlm.
> > But from your running commands, I don't see how fsdlm get involved.
> > Could you please give more detail on how to reproduce it?
>
> Sure, this is the original Pacemaker test script that failed:
>
> https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
>
> Basically it starts Corosync and DLM services and than calls:
>
> mkfs.ocfs2
> fsck.ocfs2
when I try doing your steps (btw: thanks for sharing) to run
fsck.ocfs2 it tells me:
fsck.ocfs2: Unable to access cluster service while initializing the DLM
Looking into strace right before that I see a:
connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) =
-1 ECONNREFUSED (Connection refused)
it's true I don't have ocfs2_controld running, after looking into
ocfs2-tools source code, I declare it as not compilable with recent
corosync versions.
Maybe somebody else can help so I can try to reproduce it?
btw: a "trace-cmd stream -e dlm" would be nice to debug this issue.
After I look at your debugging information it would be nice to see if
the device() read for ast callbacks are there.
- Alex
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 15:20 ` Alexander Aring
@ 2024-03-11 19:27 ` Valentin Vidić
2024-03-11 23:43 ` Alexander Aring
0 siblings, 1 reply; 14+ messages in thread
From: Valentin Vidić @ 2024-03-11 19:27 UTC (permalink / raw)
To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
On Mon, Mar 11, 2024 at 11:20:56AM -0400, Alexander Aring wrote:
> when I try doing your steps (btw: thanks for sharing) to run
> fsck.ocfs2 it tells me:
>
> fsck.ocfs2: Unable to access cluster service while initializing the DLM
>
> Looking into strace right before that I see a:
>
> connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) =
> -1 ECONNREFUSED (Connection refused)
>
> it's true I don't have ocfs2_controld running, after looking into
> ocfs2-tools source code, I declare it as not compilable with recent
> corosync versions.
Not sure about that, because I also don't have ocfs2_controld running.
This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack
should be set with:
echo pcmk > /sys/fs/ocfs2/cluster_stack
> btw: a "trace-cmd stream -e dlm" would be nice to debug this issue.
> After I look at your debugging information it would be nice to see if
> the device() read for ast callbacks are there.
Here is a trace report for the fsck call:
cpus=1
fsck.ocfs2-1696 [000] 54.621075: dlm_lock_start: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 54.692487: dlm_lock_end: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 54.695447: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 54.695636: dlm_lock_start: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 54.766959: dlm_lock_end: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 54.770765: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 54.770821: dlm_lock_start: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.845613: dlm_lock_end: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.848395: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.868951: dlm_unlock_start: ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.958319: dlm_unlock_end: ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.982397: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.982450: dlm_lock_start: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.056390: dlm_lock_end: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.058946: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.082546: dlm_unlock_start: ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.182903: dlm_unlock_end: ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.210128: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.530687: dlm_unlock_start: ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.625015: dlm_unlock_end: ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.650054: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.672033: dlm_unlock_start: ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 55.766844: dlm_unlock_end: ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 55.791829: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331
--
Valentin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 19:27 ` Valentin Vidić
@ 2024-03-11 23:43 ` Alexander Aring
2024-03-12 8:09 ` Valentin Vidić
0 siblings, 1 reply; 14+ messages in thread
From: Alexander Aring @ 2024-03-11 23:43 UTC (permalink / raw)
To: Valentin Vidić
Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
Hi,
On Mon, Mar 11, 2024 at 3:27 PM Valentin Vidić
<vvidic@valentin-vidic.from.hr> wrote:
>
> On Mon, Mar 11, 2024 at 11:20:56AM -0400, Alexander Aring wrote:
> > when I try doing your steps (btw: thanks for sharing) to run
> > fsck.ocfs2 it tells me:
> >
> > fsck.ocfs2: Unable to access cluster service while initializing the DLM
> >
> > Looking into strace right before that I see a:
> >
> > connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) =
> > -1 ECONNREFUSED (Connection refused)
> >
> > it's true I don't have ocfs2_controld running, after looking into
> > ocfs2-tools source code, I declare it as not compilable with recent
> > corosync versions.
>
> Not sure about that, because I also don't have ocfs2_controld running.
> This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack
> should be set with:
>
> echo pcmk > /sys/fs/ocfs2/cluster_stack
>
I step forward here, but still I am not able to set a cluster name. I
hardcoded it in the kernel and "it seems" to work. How do you set the
cluster name? It seems that setting the cluster name in mkfs is not
the way to go.
I am able to run fsck and I see on "dlm_tool ls" a lockspace with a
similar name you have.
The bad news is, I can't reproduce it. After fsck the lockspace
releases successfully.
I see on your debug information that it uses dlm user space locks, in
my case I am using dlm kernel locks and I am asking myself why. Can
you tell me more about your cluster configuration?
Somehow in your case fsck is using the DLM user space locks
functionality from libdlm (the user space DLM locking library) but in
my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM
kernel locks.
I am asking myself how this is supposed to work.
> > btw: a "trace-cmd stream -e dlm" would be nice to debug this issue.
> > After I look at your debugging information it would be nice to see if
> > the device() read for ast callbacks are there.
>
> Here is a trace report for the fsck call:
>
> cpus=1
> fsck.ocfs2-1696 [000] 54.621075: dlm_lock_start: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331
> fsck.ocfs2-1696 [000] 54.692487: dlm_lock_end: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
> fsck.ocfs2-1696 [000] 54.695447: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331
> fsck.ocfs2-1696 [000] 54.695636: dlm_lock_start: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
> fsck.ocfs2-1696 [000] 54.766959: dlm_lock_end: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
> fsck.ocfs2-1696 [000] 54.770765: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030
> fsck.ocfs2-1696 [000] 54.770821: dlm_lock_start: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
> fsck.ocfs2-1696 [000] 54.845613: dlm_lock_end: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
> fsck.ocfs2-1696 [000] 54.848395: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
> fsck.ocfs2-1696 [000] 54.868951: dlm_unlock_start: ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
> fsck.ocfs2-1696 [000] 54.958319: dlm_unlock_end: ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
> fsck.ocfs2-1696 [000] 54.982397: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531
> fsck.ocfs2-1696 [000] 54.982450: dlm_lock_start: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
> fsck.ocfs2-1696 [000] 55.056390: dlm_lock_end: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
> fsck.ocfs2-1696 [000] 55.058946: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
> fsck.ocfs2-1696 [000] 55.082546: dlm_unlock_start: ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
> fsck.ocfs2-1696 [000] 55.182903: dlm_unlock_end: ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
> fsck.ocfs2-1696 [000] 55.210128: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531
> fsck.ocfs2-1696 [000] 55.530687: dlm_unlock_start: ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
> fsck.ocfs2-1696 [000] 55.625015: dlm_unlock_end: ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
> fsck.ocfs2-1696 [000] 55.650054: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030
> fsck.ocfs2-1696 [000] 55.672033: dlm_unlock_start: ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331
> fsck.ocfs2-1696 [000] 55.766844: dlm_unlock_end: ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
> fsck.ocfs2-1696 [000] 55.791829: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331
>
looks okay to me. For dlm user locks there is currently an issue with
lvbarea, see [0]. Not sure if this is why you have such issues right
now, it depends what fsck is doing with the lvb.
- Alex
[0] https://lore.kernel.org/gfs2/CAK-6q+h-RngJqJ6Rx2h4a0Qy1j+ZRvCn5E-3Qee7_P3pWXJonQ@mail.gmail.com/T/#ma861c86ab54991a0d9a2742c6983544289f3bdd0
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-11 23:43 ` Alexander Aring
@ 2024-03-12 8:09 ` Valentin Vidić
2024-03-12 17:55 ` Alexander Aring
0 siblings, 1 reply; 14+ messages in thread
From: Valentin Vidić @ 2024-03-12 8:09 UTC (permalink / raw)
To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]
On Mon, Mar 11, 2024 at 07:43:41PM -0400, Alexander Aring wrote:
> I step forward here, but still I am not able to set a cluster name. I
> hardcoded it in the kernel and "it seems" to work. How do you set the
> cluster name? It seems that setting the cluster name in mkfs is not
> the way to go.
I think the cluster name comes from corosync, here is the config I have
there:
totem {
version: 2
cluster_name: debian
crypto_cipher: none
crypto_hash: none
}
logging {
fileline: off
to_stderr: yes
to_logfile: yes
logfile: /var/log/corosync/corosync.log
to_syslog: yes
debug: off
logger_subsys {
subsys: QUORUM
debug: off
}
}
quorum {
provider: corosync_votequorum
}
nodelist {
node {
name: node1
nodeid: 1
ring0_addr: 127.0.0.1
}
}
The status of the services is as follows:
# corosync-quorumtool
Quorum information
------------------
Date: Tue Mar 12 07:49:18 2024
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 1
Ring ID: 1.5
Quorate: Yes
Votequorum information
----------------------
Expected votes: 1
Highest expected: 1
Total votes: 1
Quorum: 1
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)
# dlm_tool dump_config
daemon_debug=0
foreground=1 (cli option)
log_debug=0
protocol=detect
port=21064
debug_logfile=0
bind_all=0
mark=0
enable_fscontrol=0
enable_plock=1
plock_debug=0
plock_rate_limit=0
plock_ownership=0
drop_resources_time=10000
drop_resources_count=10
drop_resources_age=10000
post_join_delay=30
enable_fencing=1
enable_concurrent_fencing=0
enable_startup_fencing=1
repeat_failed_fencing=1
enable_quorum_fencing=1
enable_quorum_lockspace=1
enable_helper=1
help=-1
version=-1
> I am able to run fsck and I see on "dlm_tool ls" a lockspace with a
> similar name you have.
>
> The bad news is, I can't reproduce it. After fsck the lockspace
> releases successfully.
>
> I see on your debug information that it uses dlm user space locks, in
> my case I am using dlm kernel locks and I am asking myself why. Can
> you tell me more about your cluster configuration?
>
> Somehow in your case fsck is using the DLM user space locks
> functionality from libdlm (the user space DLM locking library) but in
> my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM
> kernel locks.
> I am asking myself how this is supposed to work.
I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs
module is not loaded, so it falls back to user space locks?
--
Valentin
[-- Attachment #2: fsck.1654 --]
[-- Type: text/plain, Size: 55814 bytes --]
execve("/usr/sbin/fsck.ocfs2", ["fsck.ocfs2", "-f", "-y", "/dev/loop0"], 0x7fff21dd7108 /* 27 vars */) = 0
brk(NULL) = 0x563054c7d000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13065, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 13065, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f61b3401000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b33ff000
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdlm_lt.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18424, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f9000
mmap(0x7f61b33fa000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33fa000
mmap(0x7f61b33fc000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fc000
mmap(0x7f61b33fd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fd000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcmap.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=26672, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 28816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f1000
mmap(0x7f61b33f3000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33f3000
mmap(0x7f61b33f6000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f6000
mmap(0x7f61b33f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f7000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18344, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20552, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33eb000
mmap(0x7f61b33ed000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33ed000
mmap(0x7f61b33ee000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ee000
mmap(0x7f61b33ef000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ef000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libaio.so.1t64", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14184, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16432, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33e6000
mmap(0x7f61b33e7000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33e7000
mmap(0x7f61b33e8000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e8000
mmap(0x7f61b33e9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e9000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220x\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1926256, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1974096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3204000
mmap(0x7f61b322a000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b322a000
mmap(0x7f61b337f000, 344064, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17b000) = 0x7f61b337f000
mmap(0x7f61b33d3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x7f61b33d3000
mmap(0x7f61b33d9000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b33d9000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcorosync_common.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13952, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16400, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b31ff000
mmap(0x7f61b3200000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b3200000
mmap(0x7f61b3201000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3201000
mmap(0x7f61b3202000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3202000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libqb.so.100", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=165872, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b31fd000
mmap(NULL, 430816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3193000
mmap(0x7f61b319a000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f61b319a000
mmap(0x7f61b31b1000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) = 0x7f61b31b1000
mmap(0x7f61b31ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x27000) = 0x7f61b31ba000
mmap(0x7f61b31bc000, 262880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b31bc000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libsystemd.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=918624, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 923208, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30b1000
mmap(0x7f61b30c8000, 573440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f61b30c8000
mmap(0x7f61b3154000, 200704, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa3000) = 0x7f61b3154000
mmap(0x7f61b3185000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd3000) = 0x7f61b3185000
mmap(0x7f61b3192000, 1608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b3192000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200w\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47288, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 45128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30a5000
mmap(0x7f61b30a8000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b30a8000
mmap(0x7f61b30ad000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f61b30ad000
mmap(0x7f61b30af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f61b30af000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=1336728, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 1340360, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f5d000
mmap(0x7f61b2f6c000, 966656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f61b2f6c000
mmap(0x7f61b3058000, 274432, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xfb000) = 0x7f61b3058000
mmap(0x7f61b309b000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13d000) = 0x7f61b309b000
mmap(0x7f61b30a4000, 968, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b30a4000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblz4.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=149952, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 152032, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f37000
mmap(0x7f61b2f3a000, 122880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b2f3a000
mmap(0x7f61b2f58000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f61b2f58000
mmap(0x7f61b2f5b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f61b2f5b000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=247800, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2f35000
mmap(NULL, 245800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2ef8000
mmap(0x7f61b2efc000, 167936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f61b2efc000
mmap(0x7f61b2f25000, 57344, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2d000) = 0x7f61b2f25000
mmap(0x7f61b2f33000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x7f61b2f33000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=784376, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 786496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e37000
mmap(0x7f61b2e3c000, 679936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e3c000
mmap(0x7f61b2ee2000, 81920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xab000) = 0x7f61b2ee2000
mmap(0x7f61b2ef6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xbe000) = 0x7f61b2ef6000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=161864, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 164304, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e0e000
mmap(0x7f61b2e13000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e13000
mmap(0x7f61b2e2a000, 45056, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7f61b2e2a000
mmap(0x7f61b2e35000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b2e35000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e0c000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e09000
arch_prctl(ARCH_SET_FS, 0x7f61b2e09800) = 0
set_tid_address(0x7f61b2e09ad0) = 1654
set_robust_list(0x7f61b2e09ae0, 24) = 0
rseq(0x7f61b2e0a120, 0x20, 0, 0x53053053) = 0
mprotect(0x7f61b33d3000, 16384, PROT_READ) = 0
mprotect(0x7f61b2e35000, 4096, PROT_READ) = 0
mprotect(0x7f61b2ef6000, 4096, PROT_READ) = 0
mprotect(0x7f61b2f33000, 4096, PROT_READ) = 0
mprotect(0x7f61b2f5b000, 4096, PROT_READ) = 0
mprotect(0x7f61b309b000, 20480, PROT_READ) = 0
mprotect(0x7f61b30af000, 4096, PROT_READ) = 0
mprotect(0x7f61b3185000, 49152, PROT_READ) = 0
mprotect(0x7f61b31ba000, 4096, PROT_READ) = 0
mprotect(0x7f61b3202000, 4096, PROT_READ) = 0
mprotect(0x7f61b33e9000, 4096, PROT_READ) = 0
mprotect(0x7f61b33ef000, 4096, PROT_READ) = 0
mprotect(0x7f61b33f7000, 4096, PROT_READ) = 0
mprotect(0x7f61b33fd000, 4096, PROT_READ) = 0
mprotect(0x563052eed000, 4096, PROT_READ) = 0
mprotect(0x7f61b3435000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f61b3401000, 13065) = 0
getrandom("\xb3\xf1\xb9\xa1\x5c\xd8\xbe\xe2", 8, GRND_NONBLOCK) = 8
brk(NULL) = 0x563054c7d000
brk(0x563054c9e000) = 0x563054c9e000
prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
newfstatat(2, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
write(2, "fsck.ocfs2 1.8.8\n", 17) = 17
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
openat(AT_FDCWD, "/proc/swaps", O_RDONLY) = 3
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPr"..., 1024) = 39
read(3, "", 1024) = 0
close(3) = 0
openat(AT_FDCWD, "/proc/mounts", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0
newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616
newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "", 1024) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
close(3) = 0
openat(AT_FDCWD, "/etc/mtab", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0
newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616
newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "", 1024) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
close(3) = 0
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_EXCL) = 3
close(3) = 0
rt_sigaction(SIGTERM, {sa_handler=0x563052e98e30, sa_mask=[TERM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x563052e98e30, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDWR|O_DIRECT) = 3
pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512
uname({sysname="Linux", nodename="debian", ...}) = 0
ioctl(3, BLKROGET, [0]) = 0
pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512
pread64(3, "\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2"..., 512, 1024) = 512
pread64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024
openat(AT_FDCWD, "/sys/fs/ocfs2/cluster_stack", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(4, "pcmk\n", 4096) = 5
close(4) = 0
openat(AT_FDCWD, "/sys/fs/o2cb/interface_revision", O_RDONLY) = 4
read(4, "5\n", 15) = 2
read(4, "", 13) = 0
close(4) = 0
newfstatat(AT_FDCWD, "/sys/kernel/config", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
statfs("/sys/kernel/config", {f_type=CONFIGFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0x20, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 4
ioctl(4, BLKSSZGET, [512]) = 0
close(4) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\6\0\0\0\0\0@\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 14336) = 1024
openat(AT_FDCWD, "/sys/module/ocfs2_dlmfs/parameters/capabilities", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/proc/misc", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(4, "122 dlm_plock\n123 dlm-monitor\n12"..., 1024) = 220
close(4) = 0
newfstatat(AT_FDCWD, "/dev/misc/dlm-control", {st_mode=S_IFCHR|0666, st_rdev=makedev(0xa, 0x7c), ...}, 0) = 0
openat(AT_FDCWD, "/dev/misc/dlm-control", O_RDWR) = 4
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "\6\0\0\0\0\0\0\0\2\0\0\0", 12) = 12
openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5
newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(5, "1\n", 4096) = 2
close(5) = 0
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 5
read(5, "\221cK\240s\342\265\212", 8) = 8
close(5) = 0
brk(0x563054cc9000) = 0x563054cc9000
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\4\1\0\0\0\0\0\0\0\0\0\00048E8C81F"..., 152) = 121
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", 0x7ffdc4986d90, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/misc/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
newfstatat(5, "", {st_mode=S_IFDIR|0755, st_size=120, ...}, AT_EMPTY_PATH) = 0
getdents64(5, 0x563054ca9790 /* 6 entries */, 32768) = 200
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFCHR|0660, st_rdev=makedev(0xa, 0x79), ...}, 0) = 0
close(5) = 0
openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5
fchmod(5, 0600) = 0
fcntl(5, F_SETFD, FD_CLOEXEC) = 0
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\3\21\0\0\10\0\0\0\0\0\0\0\0\0\0\0"..., 137) = 1
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0h\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 104
openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 6
newfstatat(6, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(6, "1\n", 4096) = 2
close(6) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 2
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 3
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\3\0\0\0\0\0\0\0"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 4
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\4\0\0\0\0\0\0\0"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
newfstatat(1, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
write(1, "Checking OCFS2 filesystem in /de"..., 41) = 41
write(1, " Label: <NONE>\n", 29) = 29
write(1, " UUID: 48E8C81F03"..., 55) = 55
write(1, " Number of blocks: 204800\n", 29) = 29
write(1, " Block size: 1024\n", 27) = 27
write(1, " Number of clusters: 51200\n", 28) = 28
write(1, " Cluster size: 4096\n", 27) = 27
write(1, " Number of slots: 2\n\n", 25) = 25
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000
munmap(0x7f61afe00000, 50368512) = 0
mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000
mmap(NULL, 3149824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afa00000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mlock(0x7f61afe00400, 50364416) = 0
mlock(0x7f61afa00010, 3147776) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024
pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 552960) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024
pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 4747264) = 1024
brk(0x563054cea000) = 0x563054cea000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=201494528, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
munlock(0x7f61afe00400, 50364416) = 0
munmap(0x7f61afe00000, 50368512) = 0
munlock(0x7f61afa00010, 3147776) = 0
munmap(0x7f61afa00000, 3149824) = 0
mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000
munmap(0x7f61b0600000, 40300544) = 0
mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000
brk(0x563054f61000) = 0x563054f61000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=220368896, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mlock(0x7f61b0600400, 40296448) = 0
mlock(0x563054cd9650, 2518528) = 0
munlock(0x7f61b0600400, 40296448) = 0
munmap(0x7f61b0600000, 40300544) = 0
munlock(0x563054cd9650, 2518528) = 0
mmap(NULL, 20152320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b1a00000
munmap(0x7f61b1a00000, 20152320) = 0
brk(0x563056031000) = 0x563056031000
brk(0x563056165000) = 0x563056165000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=216174592, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mlock(0x563054cd9800, 20148224) = 0
mlock(0x563056010a90, 1259264) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 26624) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 27648) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024
write(1, "/dev/loop0 was run with -f, chec"..., 42) = 42
write(1, "Pass 0a: Checking cluster alloca"..., 44) = 44
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33247}, ru_stime={tv_sec=0, tv_usec=546077}, ...}) = 0
pread64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
io_setup(7, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615a800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615ac00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615bc00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615c000, aio_nbytes=1024, aio_offset=188743680}]) = 7
io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x5630561553b0, res=1024, res2=0}, {data=0, obj=0x563056155370, res=1024, res2=0}, {data=0, obj=0x563056155330, res=1024, res2=0}, {data=0, obj=0x5630561552f0, res=1024, res2=0}, {data=0, obj=0x5630561552b0, res=1024, res2=0}, {data=0, obj=0x563056155270, res=1024, res2=0}, {data=0, obj=0x563056155230, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7
io_destroy(0x7f61b3404000) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33261}, ru_stime={tv_sec=0, tv_usec=546303}, ...}) = 0
write(1, "Pass 0b: Checking inode allocati"..., 42) = 42
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33263}, ru_stime={tv_sec=0, tv_usec=546336}, ...}) = 0
pread64(3, "INODE01\0L\23\t<\377\377\4\0\0\0\0\0D\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 12288) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=8192}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
pread64(3, "INODE01\0L\23\t<\377\377\16\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 22528) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=13172736}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
brk(0x563056186000) = 0x563056186000
pread64(3, "INODE01\0L\23\t<\377\377\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 23552) = 1024
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33331}, ru_stime={tv_sec=0, tv_usec=547451}, ...}) = 0
write(1, "Pass 0c: Checking extent block a"..., 49) = 49
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33335}, ru_stime={tv_sec=0, tv_usec=547517}, ...}) = 0
pread64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=8941568}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x56305616abb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
pread64(3, "INODE01\0L\23\t<\377\377\r\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 21504) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=11038720}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33390}, ru_stime={tv_sec=0, tv_usec=548424}, ...}) = 0
write(1, "Pass 1: Checking inodes and bloc"..., 35) = 35
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33394}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
brk(0x563056597000) = 0x563056597000
pread64(3, "INODE01\0L\23\t<\377\377\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 277504, 9216) = 277504
pread64(3, "INODE01\0L\23\t<\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 13173760) = 1048576
pread64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1047552, 14222336) = 1047552
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38873}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 2: Checking directory entri"..., 35) = 35
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38975}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
io_setup(8, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 8, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176400, aio_nbytes=1024, aio_offset=9216}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176800, aio_nbytes=1024, aio_offset=18432}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176c00, aio_nbytes=1024, aio_offset=19456}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177000, aio_nbytes=1024, aio_offset=286720}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177400, aio_nbytes=1024, aio_offset=287744}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177800, aio_nbytes=1024, aio_offset=288768}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177c00, aio_nbytes=1024, aio_offset=289792}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056178000, aio_nbytes=1024, aio_offset=13173760}]) = 8
io_pgetevents(0x7f61b3404000, 8, 8, [{data=0, obj=0x563056154bb0, res=1024, res2=0}, {data=0, obj=0x563056154b70, res=1024, res2=0}, {data=0, obj=0x563056154b30, res=1024, res2=0}, {data=0, obj=0x563056154af0, res=1024, res2=0}, {data=0, obj=0x563056154ab0, res=1024, res2=0}, {data=0, obj=0x563056154a70, res=1024, res2=0}, {data=0, obj=0x563056154a30, res=1024, res2=0}, {data=0, obj=0x5630561549f0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 8
io_destroy(0x7f61b3404000) = 0
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056163400, aio_nbytes=1024, aio_offset=8192}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056157400, aio_nbytes=1024, aio_offset=13172736}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
write(1, "[DX_TREE_MISSING] Directory 10 i"..., 57) = 57
write(1, " y\n", 3) = 3
pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024
pread64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305657f800, aio_nbytes=1024, aio_offset=8941568}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
pwrite64(3, "GROUP01\0\0\1\0\10\376\7\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 8941568) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024
pwrite64(3, "DXDIR01\0\211\270\365\260\265\3\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pwrite64(3, "DXDIR01\0\237\27d\200\241\1\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pwrite64(3, "DXDIR01\0\262@\254\245\334\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\177\233\217\370\327\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\377\204g\24\336\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0u\236n\322\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\n4z\265\317\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\0b`\367\342\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\303\271\272\342\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\0172\365\275\t\t\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\233\214\21a\213\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\27\363,B\232\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\231\337}\24\31\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\200\264I|\6\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0q\6\316\373U\17\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\261C\321\320k\2\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0!\324',\362\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\343\20\n\340\232\f\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0T\214\2611\3\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\336aa\3229\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0k\274\244\3l\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0s\322D\305\275\32\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\334\304\302\352s\35\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\200+]\37\254\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\241\320\204\4=\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\25\234Q\5\5\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0i\252\342\7\31\34\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024
pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024
pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024
pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024
pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024
pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024
pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024
pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024
pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024
pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024
pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=44996}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 3: Checking directory conne"..., 40) = 40
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45065}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45123}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 4a: Checking for orphaned i"..., 38) = 38
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45237}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45351}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 4b: Checking inodes link co"..., 37) = 37
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45395}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45426}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 5: Checking quota informati"..., 35) = 35
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45474}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
brk(0x5630569a1000) = 0x5630569a1000
io_setup(7, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196c00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197c00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056198000, aio_nbytes=1024, aio_offset=188743680}]) = 7
io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x56305616ffb0, res=1024, res2=0}, {data=0, obj=0x56305616ff70, res=1024, res2=0}, {data=0, obj=0x56305616ff30, res=1024, res2=0}, {data=0, obj=0x56305616fef0, res=1024, res2=0}, {data=0, obj=0x56305616feb0, res=1024, res2=0}, {data=0, obj=0x56305616fe70, res=1024, res2=0}, {data=0, obj=0x56305616fe30, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7
io_destroy(0x7f61b3404000) = 0
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13139968) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024
pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024
pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024
pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024
pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13144064) = 2048
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13148160) = 2048
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13152256) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024
pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024
pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024
pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024
pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13156352) = 2048
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13160448) = 2048
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=55343}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "All passes succeeded.\n", 22) = 22
write(1, "\n", 1) = 1
pwrite64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 13135872) = 4096
rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\355\376\20\0\0\0\10\0\0\0\2\0\0\0\377\377\377\377"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0 \0\230\304\10\0\0\0\1\0\0\0\310\16\355\376"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFLNK|0777, st_size=39, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(5, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0xa, 0x79), ...}, AT_EMPTY_PATH) = 0
close(5) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\5\1\0\0\0\0\0\0y\0\0\0\4\0\0\0\0\0\0\0"..., 120) = -1 EBUSY (Device or resource busy)
openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5
close(5) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 5
ioctl(5, BLKSSZGET, [512]) = 0
close(5) = 0
openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5
newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(5, "1\n", 4096) = 2
close(5) = 0
rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
munlock(0x563054cd9800, 20148224) = 0
munlock(0x563056010a90, 1259264) = 0
close(3) = 0
exit_group(0) = ?
+++ exited with 0 +++
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-12 8:09 ` Valentin Vidić
@ 2024-03-12 17:55 ` Alexander Aring
2024-03-12 18:37 ` Valentin Vidić
0 siblings, 1 reply; 14+ messages in thread
From: Alexander Aring @ 2024-03-12 17:55 UTC (permalink / raw)
To: Valentin Vidić
Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
Hi,
On Tue, Mar 12, 2024 at 4:09 AM Valentin Vidić
<vvidic@valentin-vidic.from.hr> wrote:
>
> On Mon, Mar 11, 2024 at 07:43:41PM -0400, Alexander Aring wrote:
> > I step forward here, but still I am not able to set a cluster name. I
> > hardcoded it in the kernel and "it seems" to work. How do you set the
> > cluster name? It seems that setting the cluster name in mkfs is not
> > the way to go.
>
> I think the cluster name comes from corosync, here is the config I have
> there:
>
> totem {
> version: 2
> cluster_name: debian
> crypto_cipher: none
> crypto_hash: none
> }
> logging {
> fileline: off
> to_stderr: yes
> to_logfile: yes
> logfile: /var/log/corosync/corosync.log
> to_syslog: yes
> debug: off
> logger_subsys {
> subsys: QUORUM
> debug: off
> }
> }
> quorum {
> provider: corosync_votequorum
> }
> nodelist {
> node {
> name: node1
> nodeid: 1
> ring0_addr: 127.0.0.1
> }
> }
>
> The status of the services is as follows:
>
> # corosync-quorumtool
> Quorum information
> ------------------
> Date: Tue Mar 12 07:49:18 2024
> Quorum provider: corosync_votequorum
> Nodes: 1
> Node ID: 1
> Ring ID: 1.5
> Quorate: Yes
>
> Votequorum information
> ----------------------
> Expected votes: 1
> Highest expected: 1
> Total votes: 1
> Quorum: 1
> Flags: Quorate
>
> Membership information
> ----------------------
> Nodeid Votes Name
> 1 1 node1 (local)
>
> # dlm_tool dump_config
> daemon_debug=0
> foreground=1 (cli option)
> log_debug=0
> protocol=detect
> port=21064
> debug_logfile=0
> bind_all=0
> mark=0
> enable_fscontrol=0
> enable_plock=1
> plock_debug=0
> plock_rate_limit=0
> plock_ownership=0
> drop_resources_time=10000
> drop_resources_count=10
> drop_resources_age=10000
> post_join_delay=30
> enable_fencing=1
> enable_concurrent_fencing=0
> enable_startup_fencing=1
> repeat_failed_fencing=1
> enable_quorum_fencing=1
> enable_quorum_lockspace=1
> enable_helper=1
> help=-1
> version=-1
>
> > I am able to run fsck and I see on "dlm_tool ls" a lockspace with a
> > similar name you have.
> >
> > The bad news is, I can't reproduce it. After fsck the lockspace
> > releases successfully.
> >
> > I see on your debug information that it uses dlm user space locks, in
> > my case I am using dlm kernel locks and I am asking myself why. Can
> > you tell me more about your cluster configuration?
> >
> > Somehow in your case fsck is using the DLM user space locks
> > functionality from libdlm (the user space DLM locking library) but in
> > my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM
> > kernel locks.
> > I am asking myself how this is supposed to work.
>
> I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs
> module is not loaded, so it falls back to user space locks?
I was able to reproduce the issue and sent a fix out.
If you want to extend your debian testing using different approach
(not libdlm) this can be done by:
mount -t ocfs2_dlmfs none /dlm
this will somehow do some UAPI that triggers dlm kernel locking (so
far I see it). How it is intended to work and which combination works
or not? I have no idea, all I can say is that you should avoid using
the same lockspace "name" for libdlm user and kernel dlm locking. I
need to recheck if we have a check for that, that might break stuff
but should not be used in such a way anyway... and maybe in future
there is no difference between kernel/user locks.
I am curious about your testing at debian cluster, do you also test gfs2?
- Alex
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-12 17:55 ` Alexander Aring
@ 2024-03-12 18:37 ` Valentin Vidić
2024-03-12 19:55 ` Alexander Aring
0 siblings, 1 reply; 14+ messages in thread
From: Valentin Vidić @ 2024-03-12 18:37 UTC (permalink / raw)
To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
On Tue, Mar 12, 2024 at 01:55:54PM -0400, Alexander Aring wrote:
> I was able to reproduce the issue and sent a fix out.
Yay, thanks a lot for testing, I can confirm the patches fix the problem
for me!
> If you want to extend your debian testing using different approach
> (not libdlm) this can be done by:
>
> mount -t ocfs2_dlmfs none /dlm
I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs
is only relevant when used with o2cb cluster type instead of corosync.
> I am curious about your testing at debian cluster, do you also test gfs2?
Yes, a similar test for gfs2 would be this:
https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads
It is not really a cluster test since it only runs on one KVM
machine. But as a package test it works well because even for
1-node cluster all the code (userspace and kernel) still needs
to work correctly.
--
Valentin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-12 18:37 ` Valentin Vidić
@ 2024-03-12 19:55 ` Alexander Aring
2024-03-12 20:03 ` Alexander Aring
2024-03-12 20:42 ` Valentin Vidić
0 siblings, 2 replies; 14+ messages in thread
From: Alexander Aring @ 2024-03-12 19:55 UTC (permalink / raw)
To: Valentin Vidić
Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
Hi,
On Tue, Mar 12, 2024 at 2:37 PM Valentin Vidić
<vvidic@valentin-vidic.from.hr> wrote:
>
> On Tue, Mar 12, 2024 at 01:55:54PM -0400, Alexander Aring wrote:
> > I was able to reproduce the issue and sent a fix out.
>
> Yay, thanks a lot for testing, I can confirm the patches fix the problem
> for me!
>
Then you can reply to the mail with:
Tested-by: Valentin Vidić <vvidic@valentin-vidic.from.hr>
that would help to bring the fix upstream.
> > If you want to extend your debian testing using different approach
> > (not libdlm) this can be done by:
> >
> > mount -t ocfs2_dlmfs none /dlm
>
> I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs
> is only relevant when used with o2cb cluster type instead of corosync.
>
Yes, I think you are right. It is a replacement for
corosync/dlm_controld handling. It seems I had both setup working at
the same time, but it still uses kernel DLM locking.
> > I am curious about your testing at debian cluster, do you also test gfs2?
>
> Yes, a similar test for gfs2 would be this:
>
> https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads
>
> It is not really a cluster test since it only runs on one KVM
> machine. But as a package test it works well because even for
> 1-node cluster all the code (userspace and kernel) still needs
> to work correctly.
>
I see, I currently have plans/ideas to add namespace support into
kernel DLM that might help you by extending your testing. The general
idea would be that you can do something like:
...
ip netns exec node1 mount /dev/fake_shared_block /mnt/node1
ip netns exec node2 mount /dev/fake_shared_block /mnt/node2
ip netns exec node3 mount /dev/fake_shared_block /mnt/node3
...
I hope that somehow explains the idea behind it. Setup your cluster
networking with namespaces, e.g. selftests of Linux kernel is doing it
[0] and the DLM lockspaces are separated by net namespaces*. It
"should" run like a cluster with several machines, it is only to do
more testing and we don't need to synchronize tests over the network.
However it will _not_ replace real cluster testing but it is a
starting point to make testing "simpler". Somewhere is even a big fat
"software" switch you can run wireshark on it to see all cluster
traffic going on.
* In theory I am only abusing net namespaces for separate DLM
lockspaces, but I don't want to introduce another kind of namespace
now. However net namespaces fit here because it is necessary for a per
node networking setup anyway.
- Alex
[0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/net/icmp.sh
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-12 19:55 ` Alexander Aring
@ 2024-03-12 20:03 ` Alexander Aring
2024-03-12 20:42 ` Valentin Vidić
1 sibling, 0 replies; 14+ messages in thread
From: Alexander Aring @ 2024-03-12 20:03 UTC (permalink / raw)
To: Valentin Vidić
Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
Hi,
On Tue, Mar 12, 2024 at 3:55 PM Alexander Aring <aahringo@redhat.com> wrote:
>
> Hi,
>
> On Tue, Mar 12, 2024 at 2:37 PM Valentin Vidić
> <vvidic@valentin-vidic.from.hr> wrote:
> >
> > On Tue, Mar 12, 2024 at 01:55:54PM -0400, Alexander Aring wrote:
> > > I was able to reproduce the issue and sent a fix out.
> >
> > Yay, thanks a lot for testing, I can confirm the patches fix the problem
> > for me!
> >
>
> Then you can reply to the mail with:
>
> Tested-by: Valentin Vidić <vvidic@valentin-vidic.from.hr>
>
> that would help to bring the fix upstream.
>
> > > If you want to extend your debian testing using different approach
> > > (not libdlm) this can be done by:
> > >
> > > mount -t ocfs2_dlmfs none /dlm
> >
> > I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs
> > is only relevant when used with o2cb cluster type instead of corosync.
> >
>
> Yes, I think you are right. It is a replacement for
> corosync/dlm_controld handling. It seems I had both setup working at
> the same time, but it still uses kernel DLM locking.
It also requires a ocfs2-tools tool at a specific location:
/sbin/ocfs2_hb_ctl
maybe this is your issue why it wasn't working. My kernel was
complaining about it because it was missing in /sbin/
- Alex
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: ocfs2 mount error
2024-03-12 19:55 ` Alexander Aring
2024-03-12 20:03 ` Alexander Aring
@ 2024-03-12 20:42 ` Valentin Vidić
1 sibling, 0 replies; 14+ messages in thread
From: Valentin Vidić @ 2024-03-12 20:42 UTC (permalink / raw)
To: Alexander Aring; +Cc: Joseph Qi, ocfs2-devel, David Teigland, gfs2, ocfs2-devel
On Tue, Mar 12, 2024 at 03:55:46PM -0400, Alexander Aring wrote:
> Then you can reply to the mail with:
>
> Tested-by: Valentin Vidić <vvidic@valentin-vidic.from.hr>
>
> that would help to bring the fix upstream.
Sure, replied to the other thread.
> I see, I currently have plans/ideas to add namespace support into
> kernel DLM that might help you by extending your testing. The general
> idea would be that you can do something like:
>
> ...
> ip netns exec node1 mount /dev/fake_shared_block /mnt/node1
> ip netns exec node2 mount /dev/fake_shared_block /mnt/node2
> ip netns exec node3 mount /dev/fake_shared_block /mnt/node3
> ...
Yeah, that would definitely be cool. And thanks for the pointers,
I will definitely try to use namespaces for some simpler corosync
tests first.
> It also requires a ocfs2-tools tool at a specific location:
>
> /sbin/ocfs2_hb_ctl
>
> maybe this is your issue why it wasn't working. My kernel was
> complaining about it because it was missing in /sbin/
That binary is already in the Debian package. I think the error
fsck reported was something like "Could not create domain".
--
Valentin
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2024-03-12 20:42 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-10 21:46 ocfs2 mount error Valentin Vidić
2024-03-11 8:37 ` Joseph Qi
2024-03-11 9:02 ` Valentin Vidić
2024-03-11 9:27 ` Joseph Qi
2024-03-11 12:28 ` Heming Zhao
2024-03-11 15:20 ` Alexander Aring
2024-03-11 19:27 ` Valentin Vidić
2024-03-11 23:43 ` Alexander Aring
2024-03-12 8:09 ` Valentin Vidić
2024-03-12 17:55 ` Alexander Aring
2024-03-12 18:37 ` Valentin Vidić
2024-03-12 19:55 ` Alexander Aring
2024-03-12 20:03 ` Alexander Aring
2024-03-12 20:42 ` Valentin Vidić
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox