* [Ocfs2-devel] report BUG: io_uring triggers umount error @ 2023-02-24 0:32 Heming Zhao via Ocfs2-devel 2023-02-24 6:54 ` Joseph Qi via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Heming Zhao via Ocfs2-devel @ 2023-02-24 0:32 UTC (permalink / raw) To: joseph.qi, ocfs2-devel Hello List, I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time he/she could fix it. This bug is blocking fstest generic/013 test case, and also blocking fstest to do later test cases. How to trigger: ``` git clone git://git.kernel.dk/liburing.git cd liburing make cd examples mount -t ocfs2 /dev/sda /mnt cp /etc/hosts /mnt/a ./link-cp /mnt/a /mnt/b umount /mnt ``` umount trigger error message: ``` # umount /mnt umount: /mnt: target is busy. ``` The umount error can only be triggered by liburing write operation. Thanks, Heming _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-24 0:32 [Ocfs2-devel] report BUG: io_uring triggers umount error Heming Zhao via Ocfs2-devel @ 2023-02-24 6:54 ` Joseph Qi via Ocfs2-devel 2023-02-24 7:48 ` Heming Zhao via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Joseph Qi via Ocfs2-devel @ 2023-02-24 6:54 UTC (permalink / raw) To: Heming Zhao, ocfs2-devel I can reproduce this in my local VM. I've traced ocfs2_dismount_volume and found that it hasn't been called. So EBUSY is returned in VFS layer. I guess something wrong when doing a copy with linked SQEs (normal copy seems no problem). Thanks, Joseph On 2/24/23 8:32 AM, Heming Zhao wrote: > Hello List, > > I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time > he/she could fix it. This bug is blocking fstest generic/013 test case, and also > blocking fstest to do later test cases. > > How to trigger: > ``` > git clone git://git.kernel.dk/liburing.git > cd liburing > make > cd examples > mount -t ocfs2 /dev/sda /mnt > cp /etc/hosts /mnt/a > ./link-cp /mnt/a /mnt/b > umount /mnt > ``` > > umount trigger error message: > ``` > # umount /mnt > umount: /mnt: target is busy. > ``` > > The umount error can only be triggered by liburing write operation. > > Thanks, > Heming _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-24 6:54 ` Joseph Qi via Ocfs2-devel @ 2023-02-24 7:48 ` Heming Zhao via Ocfs2-devel 2023-02-24 7:52 ` Joseph Qi via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Heming Zhao via Ocfs2-devel @ 2023-02-24 7:48 UTC (permalink / raw) To: Joseph Qi, ocfs2-devel On 2/24/23 2:54 PM, Joseph Qi wrote: > I can reproduce this in my local VM. > I've traced ocfs2_dismount_volume and found that it hasn't been called. > So EBUSY is returned in VFS layer. I guess something wrong when doing > a copy with linked SQEs (normal copy seems no problem). > I am inclined to agree with you. I also test liburing examples apps on ext4 partition, everything looks fine. I used below bpftrace method, the retval is '3'. bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') Thanks, Heming > > On 2/24/23 8:32 AM, Heming Zhao wrote: >> Hello List, >> >> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time >> he/she could fix it. This bug is blocking fstest generic/013 test case, and also >> blocking fstest to do later test cases. >> >> How to trigger: >> ``` >> git clone git://git.kernel.dk/liburing.git >> cd liburing >> make >> cd examples >> mount -t ocfs2 /dev/sda /mnt >> cp /etc/hosts /mnt/a >> ./link-cp /mnt/a /mnt/b >> umount /mnt >> ``` >> >> umount trigger error message: >> ``` >> # umount /mnt >> umount: /mnt: target is busy. >> ``` >> >> The umount error can only be triggered by liburing write operation. >> >> Thanks, >> Heming _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-24 7:48 ` Heming Zhao via Ocfs2-devel @ 2023-02-24 7:52 ` Joseph Qi via Ocfs2-devel 2023-02-24 8:03 ` Heming Zhao via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Joseph Qi via Ocfs2-devel @ 2023-02-24 7:52 UTC (permalink / raw) To: Heming Zhao, Joseph Qi, ocfs2-devel On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: > On 2/24/23 2:54 PM, Joseph Qi wrote: >> I can reproduce this in my local VM. >> I've traced ocfs2_dismount_volume and found that it hasn't been called. >> So EBUSY is returned in VFS layer. I guess something wrong when doing >> a copy with linked SQEs (normal copy seems no problem). >> > > I am inclined to agree with you. I also test liburing examples apps > on ext4 partition, everything looks fine. > > I used below bpftrace method, the retval is '3'. > bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' > > It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') > Yes, that's the place return EBUSY. So the problem seems to be getmnt/putmnt not match in this case. Thanks, Joseph > >> >> On 2/24/23 8:32 AM, Heming Zhao wrote: >>> Hello List, >>> >>> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time >>> he/she could fix it. This bug is blocking fstest generic/013 test case, and also >>> blocking fstest to do later test cases. >>> >>> How to trigger: >>> ``` >>> git clone git://git.kernel.dk/liburing.git >>> cd liburing >>> make >>> cd examples >>> mount -t ocfs2 /dev/sda /mnt >>> cp /etc/hosts /mnt/a >>> ./link-cp /mnt/a /mnt/b >>> umount /mnt >>> ``` >>> >>> umount trigger error message: >>> ``` >>> # umount /mnt >>> umount: /mnt: target is busy. >>> ``` >>> >>> The umount error can only be triggered by liburing write operation. >>> >>> Thanks, >>> Heming > > > _______________________________________________ > Ocfs2-devel mailing list > Ocfs2-devel@oss.oracle.com > https://oss.oracle.com/mailman/listinfo/ocfs2-devel _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-24 7:52 ` Joseph Qi via Ocfs2-devel @ 2023-02-24 8:03 ` Heming Zhao via Ocfs2-devel 2023-02-24 10:59 ` Joseph Qi via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Heming Zhao via Ocfs2-devel @ 2023-02-24 8:03 UTC (permalink / raw) To: Joseph Qi, Joseph Qi, ocfs2-devel On 2/24/23 3:52 PM, Joseph Qi wrote: > > > On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: >> On 2/24/23 2:54 PM, Joseph Qi wrote: >>> I can reproduce this in my local VM. >>> I've traced ocfs2_dismount_volume and found that it hasn't been called. >>> So EBUSY is returned in VFS layer. I guess something wrong when doing >>> a copy with linked SQEs (normal copy seems no problem). >>> >> >> I am inclined to agree with you. I also test liburing examples apps >> on ext4 partition, everything looks fine. >> >> I used below bpftrace method, the retval is '3'. >> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' >> >> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') >> > Yes, that's the place return EBUSY. > So the problem seems to be getmnt/putmnt not match in this case. > I didn't familiar with setting up kernel bi-search env. I used one last year openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. So there is a possibility one ocfs2 commit introduced this issue. Thanks, Heming > >> >>> >>> On 2/24/23 8:32 AM, Heming Zhao wrote: >>>> Hello List, >>>> >>>> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time >>>> he/she could fix it. This bug is blocking fstest generic/013 test case, and also >>>> blocking fstest to do later test cases. >>>> >>>> How to trigger: >>>> ``` >>>> git clone git://git.kernel.dk/liburing.git >>>> cd liburing >>>> make >>>> cd examples >>>> mount -t ocfs2 /dev/sda /mnt >>>> cp /etc/hosts /mnt/a >>>> ./link-cp /mnt/a /mnt/b >>>> umount /mnt >>>> ``` >>>> >>>> umount trigger error message: >>>> ``` >>>> # umount /mnt >>>> umount: /mnt: target is busy. >>>> ``` >>>> >>>> The umount error can only be triggered by liburing write operation. >>>> >>>> Thanks, >>>> Heming >> >> >> _______________________________________________ >> Ocfs2-devel mailing list >> Ocfs2-devel@oss.oracle.com >> https://oss.oracle.com/mailman/listinfo/ocfs2-devel _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-24 8:03 ` Heming Zhao via Ocfs2-devel @ 2023-02-24 10:59 ` Joseph Qi via Ocfs2-devel 2023-02-26 8:45 ` Heming Zhao via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Joseph Qi via Ocfs2-devel @ 2023-02-24 10:59 UTC (permalink / raw) To: Heming Zhao, Joseph Qi, ocfs2-devel On 2/24/23 4:03 PM, Heming Zhao wrote: > On 2/24/23 3:52 PM, Joseph Qi wrote: >> >> >> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: >>> On 2/24/23 2:54 PM, Joseph Qi wrote: >>>> I can reproduce this in my local VM. >>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. >>>> So EBUSY is returned in VFS layer. I guess something wrong when doing >>>> a copy with linked SQEs (normal copy seems no problem). >>>> >>> >>> I am inclined to agree with you. I also test liburing examples apps >>> on ext4 partition, everything looks fine. >>> >>> I used below bpftrace method, the retval is '3'. >>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' >>> >>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') >>> >> Yes, that's the place return EBUSY. >> So the problem seems to be getmnt/putmnt not match in this case. >> > > I didn't familiar with setting up kernel bi-search env. I used one last year > openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. > So there is a possibility one ocfs2 commit introduced this issue. >You can checkout each mailine version like Linux 6.0, 6.1, ... and try to check if it can be reproduced. I've tried trace mntget/mntput using the following bpftrace script, link-cp output shows it misses a fput. #include <linux/mount.h> #include <linux/string.h> kprobe:mntget { $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; if (!strncmp(str($n), "ocfs2", 5)) { @[comm] += 1; printf("%s", kstack); } } kprobe:mntput { $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; if (!strncmp(str($n), "ocfs2", 5)) { @[comm] +=1; printf("%s", kstack); } } Joseph _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-24 10:59 ` Joseph Qi via Ocfs2-devel @ 2023-02-26 8:45 ` Heming Zhao via Ocfs2-devel 2023-02-26 15:22 ` Heming Zhao via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Heming Zhao via Ocfs2-devel @ 2023-02-26 8:45 UTC (permalink / raw) To: Joseph Qi; +Cc: ocfs2-devel On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote: > > > On 2/24/23 4:03 PM, Heming Zhao wrote: > > On 2/24/23 3:52 PM, Joseph Qi wrote: > >> > >> > >> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: > >>> On 2/24/23 2:54 PM, Joseph Qi wrote: > >>>> I can reproduce this in my local VM. > >>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. > >>>> So EBUSY is returned in VFS layer. I guess something wrong when doing > >>>> a copy with linked SQEs (normal copy seems no problem). > >>>> > >>> > >>> I am inclined to agree with you. I also test liburing examples apps > >>> on ext4 partition, everything looks fine. > >>> > >>> I used below bpftrace method, the retval is '3'. > >>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' > >>> > >>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') > >>> > >> Yes, that's the place return EBUSY. > >> So the problem seems to be getmnt/putmnt not match in this case. > >> > > > > I didn't familiar with setting up kernel bi-search env. I used one last year > > openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. > > So there is a possibility one ocfs2 commit introduced this issue. >You can checkout each mailine version like Linux 6.0, 6.1, ... and try > to check if it can be reproduced. I drop this method, my machine cpu is old, compiling kernel will take too much time. > > I've tried trace mntget/mntput using the following bpftrace script, > link-cp output shows it misses a fput. > > #include <linux/mount.h> > #include <linux/string.h> > > kprobe:mntget > { > $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; > > if (!strncmp(str($n), "ocfs2", 5)) { > @[comm] += 1; > > printf("%s", kstack); > } > } > > kprobe:mntput > { > $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; > > if (!strncmp(str($n), "ocfs2", 5)) { > @[comm] +=1; > > printf("%s", kstack); > } > } > I used below script to find source file hold f_count. e.g: ./link-cp /mnt/aa.bin /mnt/1 The script will show f_count of aa.bin only down to 3. We expect f_count value down to 1 then trigger mntput. ``` #include <linux/mount.h> #include <linux/string.h> #include <linux/dcache.h> #include <linux/path.h> #include <linux/fs_struct.h> #include <linux/fs.h> #include <linux/io_uring_types.h> k:fput { $f = (struct file *)arg0; $cnt = $f->f_count; if (!strncmp(comm, "link-cp", 7)) { printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n", str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt)); //printf("%s\n", kstack); } } ``` - Heming _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-26 8:45 ` Heming Zhao via Ocfs2-devel @ 2023-02-26 15:22 ` Heming Zhao via Ocfs2-devel 2023-02-27 2:59 ` Joseph Qi via Ocfs2-devel 0 siblings, 1 reply; 12+ messages in thread From: Heming Zhao via Ocfs2-devel @ 2023-02-26 15:22 UTC (permalink / raw) To: Joseph Qi, ocfs2-devel On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote: > On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote: > > > > > > On 2/24/23 4:03 PM, Heming Zhao wrote: > > > On 2/24/23 3:52 PM, Joseph Qi wrote: > > >> > > >> > > >> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: > > >>> On 2/24/23 2:54 PM, Joseph Qi wrote: > > >>>> I can reproduce this in my local VM. > > >>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. > > >>>> So EBUSY is returned in VFS layer. I guess something wrong when doing > > >>>> a copy with linked SQEs (normal copy seems no problem). > > >>>> > > >>> > > >>> I am inclined to agree with you. I also test liburing examples apps > > >>> on ext4 partition, everything looks fine. > > >>> > > >>> I used below bpftrace method, the retval is '3'. > > >>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' > > >>> > > >>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') > > >>> > > >> Yes, that's the place return EBUSY. > > >> So the problem seems to be getmnt/putmnt not match in this case. > > >> > > > > > > I didn't familiar with setting up kernel bi-search env. I used one last year > > > openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. > > > So there is a possibility one ocfs2 commit introduced this issue. > >You can checkout each mailine version like Linux 6.0, 6.1, ... and try > > to check if it can be reproduced. > > I drop this method, my machine cpu is old, compiling kernel will take too much > time. > > > > > I've tried trace mntget/mntput using the following bpftrace script, > > link-cp output shows it misses a fput. > > > > #include <linux/mount.h> > > #include <linux/string.h> > > > > kprobe:mntget > > { > > $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; > > > > if (!strncmp(str($n), "ocfs2", 5)) { > > @[comm] += 1; > > > > printf("%s", kstack); > > } > > } > > > > kprobe:mntput > > { > > $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; > > > > if (!strncmp(str($n), "ocfs2", 5)) { > > @[comm] +=1; > > > > printf("%s", kstack); > > } > > } > > > > I used below script to find source file hold f_count. > > e.g: ./link-cp /mnt/aa.bin /mnt/1 > The script will show f_count of aa.bin only down to 3. We expect f_count value > down to 1 then trigger mntput. > > ``` > #include <linux/mount.h> > #include <linux/string.h> > #include <linux/dcache.h> > #include <linux/path.h> > #include <linux/fs_struct.h> > #include <linux/fs.h> > #include <linux/io_uring_types.h> > > k:fput > { > $f = (struct file *)arg0; > $cnt = $f->f_count; > > if (!strncmp(comm, "link-cp", 7)) { > printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n", > str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt)); > //printf("%s\n", kstack); > } > } > ``` > I am not familiar with io_uring, and can't make sure my analysis is correct. Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure issue. io_issue_sqe + io_assign_file //call fget(), but missing fput() + def->issue() | io_read | + io_iter_do_read | | ocfs2_file_read_iter | | return: -EOPNOTSUPP //iocb->ki_flags:0x8 | + kiocb_done | + io_rw_done(&rw->kiocb, ret) | | kiocb->ki_complete(kiocb, ret) | | io_complete_rw | | __io_complete_rw_common | | + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; | | + return true | + io_req_task_queue_reissue/io_req_task_queue_fail | + return IOU_ISSUE_SKIP_COMPLETE; | + return 0; - Heming _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-26 15:22 ` Heming Zhao via Ocfs2-devel @ 2023-02-27 2:59 ` Joseph Qi via Ocfs2-devel 2023-02-28 4:50 ` Joseph Qi via Ocfs2-devel 2023-03-01 2:06 ` Joseph Qi via Ocfs2-devel 0 siblings, 2 replies; 12+ messages in thread From: Joseph Qi via Ocfs2-devel @ 2023-02-27 2:59 UTC (permalink / raw) To: Heming Zhao, Joseph Qi, ocfs2-devel On 2/26/23 11:22 PM, Heming Zhao wrote: > On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote: >> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote: >>> >>> >>> On 2/24/23 4:03 PM, Heming Zhao wrote: >>>> On 2/24/23 3:52 PM, Joseph Qi wrote: >>>>> >>>>> >>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: >>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote: >>>>>>> I can reproduce this in my local VM. >>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. >>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing >>>>>>> a copy with linked SQEs (normal copy seems no problem). >>>>>>> >>>>>> >>>>>> I am inclined to agree with you. I also test liburing examples apps >>>>>> on ext4 partition, everything looks fine. >>>>>> >>>>>> I used below bpftrace method, the retval is '3'. >>>>>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' >>>>>> >>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') >>>>>> >>>>> Yes, that's the place return EBUSY. >>>>> So the problem seems to be getmnt/putmnt not match in this case. >>>>> >>>> >>>> I didn't familiar with setting up kernel bi-search env. I used one last year >>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. >>>> So there is a possibility one ocfs2 commit introduced this issue. >>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try >>> to check if it can be reproduced. >> >> I drop this method, my machine cpu is old, compiling kernel will take too much >> time. >> >>> >>> I've tried trace mntget/mntput using the following bpftrace script, >>> link-cp output shows it misses a fput. >>> >>> #include <linux/mount.h> >>> #include <linux/string.h> >>> >>> kprobe:mntget >>> { >>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>> >>> if (!strncmp(str($n), "ocfs2", 5)) { >>> @[comm] += 1; >>> >>> printf("%s", kstack); >>> } >>> } >>> >>> kprobe:mntput >>> { >>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>> >>> if (!strncmp(str($n), "ocfs2", 5)) { >>> @[comm] +=1; >>> >>> printf("%s", kstack); >>> } >>> } >>> >> >> I used below script to find source file hold f_count. >> >> e.g: ./link-cp /mnt/aa.bin /mnt/1 >> The script will show f_count of aa.bin only down to 3. We expect f_count value >> down to 1 then trigger mntput. >> >> ``` >> #include <linux/mount.h> >> #include <linux/string.h> >> #include <linux/dcache.h> >> #include <linux/path.h> >> #include <linux/fs_struct.h> >> #include <linux/fs.h> >> #include <linux/io_uring_types.h> >> >> k:fput >> { >> $f = (struct file *)arg0; >> $cnt = $f->f_count; >> >> if (!strncmp(comm, "link-cp", 7)) { >> printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n", >> str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt)); >> //printf("%s\n", kstack); >> } >> } >> ``` >> > > I am not familiar with io_uring, and can't make sure my analysis is correct. > Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails > io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure > issue. > > io_issue_sqe > + io_assign_file //call fget(), but missing fput() > + def->issue() > | io_read > | + io_iter_do_read > | | ocfs2_file_read_iter > | | return: -EOPNOTSUPP //iocb->ki_flags:0x8 > | + kiocb_done > | + io_rw_done(&rw->kiocb, ret) > | | kiocb->ki_complete(kiocb, ret) > | | io_complete_rw > | | __io_complete_rw_common > | | + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; > | | + return true > | + io_req_task_queue_reissue/io_req_task_queue_fail > | + return IOU_ISSUE_SKIP_COMPLETE; > | > + return 0; > Seems not the case. Reissue will queue work to iowq, and fput will finally get called in io_wq_free_work() once work is handled. Thanks, Joseph _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-27 2:59 ` Joseph Qi via Ocfs2-devel @ 2023-02-28 4:50 ` Joseph Qi via Ocfs2-devel 2023-03-01 2:06 ` Joseph Qi via Ocfs2-devel 1 sibling, 0 replies; 12+ messages in thread From: Joseph Qi via Ocfs2-devel @ 2023-02-28 4:50 UTC (permalink / raw) To: Joseph Qi, Heming Zhao, ocfs2-devel On 2/27/23 10:59 AM, Joseph Qi wrote: > > > On 2/26/23 11:22 PM, Heming Zhao wrote: >> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote: >>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote: >>>> >>>> >>>> On 2/24/23 4:03 PM, Heming Zhao wrote: >>>>> On 2/24/23 3:52 PM, Joseph Qi wrote: >>>>>> >>>>>> >>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: >>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote: >>>>>>>> I can reproduce this in my local VM. >>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. >>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing >>>>>>>> a copy with linked SQEs (normal copy seems no problem). >>>>>>>> >>>>>>> >>>>>>> I am inclined to agree with you. I also test liburing examples apps >>>>>>> on ext4 partition, everything looks fine. >>>>>>> >>>>>>> I used below bpftrace method, the retval is '3'. >>>>>>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' >>>>>>> >>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') >>>>>>> >>>>>> Yes, that's the place return EBUSY. >>>>>> So the problem seems to be getmnt/putmnt not match in this case. >>>>>> >>>>> >>>>> I didn't familiar with setting up kernel bi-search env. I used one last year >>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. >>>>> So there is a possibility one ocfs2 commit introduced this issue. >>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try >>>> to check if it can be reproduced. >>> >>> I drop this method, my machine cpu is old, compiling kernel will take too much >>> time. >>> >>>> >>>> I've tried trace mntget/mntput using the following bpftrace script, >>>> link-cp output shows it misses a fput. >>>> >>>> #include <linux/mount.h> >>>> #include <linux/string.h> >>>> >>>> kprobe:mntget >>>> { >>>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>>> >>>> if (!strncmp(str($n), "ocfs2", 5)) { >>>> @[comm] += 1; >>>> >>>> printf("%s", kstack); >>>> } >>>> } >>>> >>>> kprobe:mntput >>>> { >>>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>>> >>>> if (!strncmp(str($n), "ocfs2", 5)) { >>>> @[comm] +=1; >>>> >>>> printf("%s", kstack); >>>> } >>>> } >>>> >>> >>> I used below script to find source file hold f_count. >>> >>> e.g: ./link-cp /mnt/aa.bin /mnt/1 >>> The script will show f_count of aa.bin only down to 3. We expect f_count value >>> down to 1 then trigger mntput. >>> >>> ``` >>> #include <linux/mount.h> >>> #include <linux/string.h> >>> #include <linux/dcache.h> >>> #include <linux/path.h> >>> #include <linux/fs_struct.h> >>> #include <linux/fs.h> >>> #include <linux/io_uring_types.h> >>> >>> k:fput >>> { >>> $f = (struct file *)arg0; >>> $cnt = $f->f_count; >>> >>> if (!strncmp(comm, "link-cp", 7)) { >>> printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n", >>> str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt)); >>> //printf("%s\n", kstack); >>> } >>> } >>> ``` >>> >> >> I am not familiar with io_uring, and can't make sure my analysis is correct. >> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails >> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure >> issue. >> >> io_issue_sqe >> + io_assign_file //call fget(), but missing fput() >> + def->issue() >> | io_read >> | + io_iter_do_read >> | | ocfs2_file_read_iter >> | | return: -EOPNOTSUPP //iocb->ki_flags:0x8 >> | + kiocb_done >> | + io_rw_done(&rw->kiocb, ret) >> | | kiocb->ki_complete(kiocb, ret) >> | | io_complete_rw >> | | __io_complete_rw_common >> | | + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; >> | | + return true >> | + io_req_task_queue_reissue/io_req_task_queue_fail >> | + return IOU_ISSUE_SKIP_COMPLETE; >> | >> + return 0; >> > Seems not the case. > Reissue will queue work to iowq, and fput will finally get called in > io_wq_free_work() once work is handled. > I think I've found the root cause, will send a patch later. Thanks, Joseph _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-02-27 2:59 ` Joseph Qi via Ocfs2-devel 2023-02-28 4:50 ` Joseph Qi via Ocfs2-devel @ 2023-03-01 2:06 ` Joseph Qi via Ocfs2-devel 2023-03-01 9:49 ` Heming Zhao via Ocfs2-devel 1 sibling, 1 reply; 12+ messages in thread From: Joseph Qi via Ocfs2-devel @ 2023-03-01 2:06 UTC (permalink / raw) To: Heming Zhao, Joseph Qi, ocfs2-devel Hi, On 2/27/23 10:59 AM, Joseph Qi via Ocfs2-devel wrote: > > > On 2/26/23 11:22 PM, Heming Zhao wrote: >> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote: >>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote: >>>> >>>> >>>> On 2/24/23 4:03 PM, Heming Zhao wrote: >>>>> On 2/24/23 3:52 PM, Joseph Qi wrote: >>>>>> >>>>>> >>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: >>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote: >>>>>>>> I can reproduce this in my local VM. >>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. >>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing >>>>>>>> a copy with linked SQEs (normal copy seems no problem). >>>>>>>> >>>>>>> >>>>>>> I am inclined to agree with you. I also test liburing examples apps >>>>>>> on ext4 partition, everything looks fine. >>>>>>> >>>>>>> I used below bpftrace method, the retval is '3'. >>>>>>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' >>>>>>> >>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') >>>>>>> >>>>>> Yes, that's the place return EBUSY. >>>>>> So the problem seems to be getmnt/putmnt not match in this case. >>>>>> >>>>> >>>>> I didn't familiar with setting up kernel bi-search env. I used one last year >>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. >>>>> So there is a possibility one ocfs2 commit introduced this issue. >>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try >>>> to check if it can be reproduced. >>> >>> I drop this method, my machine cpu is old, compiling kernel will take too much >>> time. >>> >>>> >>>> I've tried trace mntget/mntput using the following bpftrace script, >>>> link-cp output shows it misses a fput. >>>> >>>> #include <linux/mount.h> >>>> #include <linux/string.h> >>>> >>>> kprobe:mntget >>>> { >>>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>>> >>>> if (!strncmp(str($n), "ocfs2", 5)) { >>>> @[comm] += 1; >>>> >>>> printf("%s", kstack); >>>> } >>>> } >>>> >>>> kprobe:mntput >>>> { >>>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>>> >>>> if (!strncmp(str($n), "ocfs2", 5)) { >>>> @[comm] +=1; >>>> >>>> printf("%s", kstack); >>>> } >>>> } >>>> >>> >>> I used below script to find source file hold f_count. >>> >>> e.g: ./link-cp /mnt/aa.bin /mnt/1 >>> The script will show f_count of aa.bin only down to 3. We expect f_count value >>> down to 1 then trigger mntput. >>> >>> ``` >>> #include <linux/mount.h> >>> #include <linux/string.h> >>> #include <linux/dcache.h> >>> #include <linux/path.h> >>> #include <linux/fs_struct.h> >>> #include <linux/fs.h> >>> #include <linux/io_uring_types.h> >>> >>> k:fput >>> { >>> $f = (struct file *)arg0; >>> $cnt = $f->f_count; >>> >>> if (!strncmp(comm, "link-cp", 7)) { >>> printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n", >>> str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt)); >>> //printf("%s\n", kstack); >>> } >>> } >>> ``` >>> >> >> I am not familiar with io_uring, and can't make sure my analysis is correct. >> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails >> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure >> issue. >> >> io_issue_sqe >> + io_assign_file //call fget(), but missing fput() >> + def->issue() >> | io_read >> | + io_iter_do_read >> | | ocfs2_file_read_iter >> | | return: -EOPNOTSUPP //iocb->ki_flags:0x8 >> | + kiocb_done >> | + io_rw_done(&rw->kiocb, ret) >> | | kiocb->ki_complete(kiocb, ret) >> | | io_complete_rw >> | | __io_complete_rw_common >> | | + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; >> | | + return true >> | + io_req_task_queue_reissue/io_req_task_queue_fail >> | + return IOU_ISSUE_SKIP_COMPLETE; >> | >> + return 0; >> > Seems not the case. > Reissue will queue work to iowq, and fput will finally get called in > io_wq_free_work() once work is handled. > I've sent a fix to io_uring, please refer: https://lore.kernel.org/io-uring/167758917178.12826.13481934362166793957.b4-ty@kernel.dk/T/#t Thanks, Joseph _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Ocfs2-devel] report BUG: io_uring triggers umount error 2023-03-01 2:06 ` Joseph Qi via Ocfs2-devel @ 2023-03-01 9:49 ` Heming Zhao via Ocfs2-devel 0 siblings, 0 replies; 12+ messages in thread From: Heming Zhao via Ocfs2-devel @ 2023-03-01 9:49 UTC (permalink / raw) To: Joseph Qi, Joseph Qi, ocfs2-devel On 3/1/23 10:06 AM, Joseph Qi wrote: > Hi, > > On 2/27/23 10:59 AM, Joseph Qi via Ocfs2-devel wrote: >> >> >> On 2/26/23 11:22 PM, Heming Zhao wrote: >>> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote: >>>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote: >>>>> >>>>> >>>>> On 2/24/23 4:03 PM, Heming Zhao wrote: >>>>>> On 2/24/23 3:52 PM, Joseph Qi wrote: >>>>>>> >>>>>>> >>>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote: >>>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote: >>>>>>>>> I can reproduce this in my local VM. >>>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called. >>>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing >>>>>>>>> a copy with linked SQEs (normal copy seems no problem). >>>>>>>>> >>>>>>>> >>>>>>>> I am inclined to agree with you. I also test liburing examples apps >>>>>>>> on ext4 partition, everything looks fine. >>>>>>>> >>>>>>>> I used below bpftrace method, the retval is '3'. >>>>>>>> bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}' >>>>>>>> >>>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3') >>>>>>>> >>>>>>> Yes, that's the place return EBUSY. >>>>>>> So the problem seems to be getmnt/putmnt not match in this case. >>>>>>> >>>>>> >>>>>> I didn't familiar with setting up kernel bi-search env. I used one last year >>>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist. >>>>>> So there is a possibility one ocfs2 commit introduced this issue. >>>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try >>>>> to check if it can be reproduced. >>>> >>>> I drop this method, my machine cpu is old, compiling kernel will take too much >>>> time. >>>> >>>>> >>>>> I've tried trace mntget/mntput using the following bpftrace script, >>>>> link-cp output shows it misses a fput. >>>>> >>>>> #include <linux/mount.h> >>>>> #include <linux/string.h> >>>>> >>>>> kprobe:mntget >>>>> { >>>>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>>>> >>>>> if (!strncmp(str($n), "ocfs2", 5)) { >>>>> @[comm] += 1; >>>>> >>>>> printf("%s", kstack); >>>>> } >>>>> } >>>>> >>>>> kprobe:mntput >>>>> { >>>>> $n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name; >>>>> >>>>> if (!strncmp(str($n), "ocfs2", 5)) { >>>>> @[comm] +=1; >>>>> >>>>> printf("%s", kstack); >>>>> } >>>>> } >>>>> >>>> >>>> I used below script to find source file hold f_count. >>>> >>>> e.g: ./link-cp /mnt/aa.bin /mnt/1 >>>> The script will show f_count of aa.bin only down to 3. We expect f_count value >>>> down to 1 then trigger mntput. >>>> >>>> ``` >>>> #include <linux/mount.h> >>>> #include <linux/string.h> >>>> #include <linux/dcache.h> >>>> #include <linux/path.h> >>>> #include <linux/fs_struct.h> >>>> #include <linux/fs.h> >>>> #include <linux/io_uring_types.h> >>>> >>>> k:fput >>>> { >>>> $f = (struct file *)arg0; >>>> $cnt = $f->f_count; >>>> >>>> if (!strncmp(comm, "link-cp", 7)) { >>>> printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n", >>>> str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt)); >>>> //printf("%s\n", kstack); >>>> } >>>> } >>>> ``` >>>> >>> >>> I am not familiar with io_uring, and can't make sure my analysis is correct. >>> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails >>> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure >>> issue. >>> >>> io_issue_sqe >>> + io_assign_file //call fget(), but missing fput() >>> + def->issue() >>> | io_read >>> | + io_iter_do_read >>> | | ocfs2_file_read_iter >>> | | return: -EOPNOTSUPP //iocb->ki_flags:0x8 >>> | + kiocb_done >>> | + io_rw_done(&rw->kiocb, ret) >>> | | kiocb->ki_complete(kiocb, ret) >>> | | io_complete_rw >>> | | __io_complete_rw_common >>> | | + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; >>> | | + return true >>> | + io_req_task_queue_reissue/io_req_task_queue_fail >>> | + return IOU_ISSUE_SKIP_COMPLETE; >>> | >>> + return 0; >>> >> Seems not the case. >> Reissue will queue work to iowq, and fput will finally get called in >> io_wq_free_work() once work is handled. >> > I've sent a fix to io_uring, please refer: > https://lore.kernel.org/io-uring/167758917178.12826.13481934362166793957.b4-ty@kernel.dk/T/#t > > Thanks, > Joseph > Got it. Thank you for your quickly fix. Let's make ocfs2 more powerful/stable. - Heming _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2023-03-01 9:49 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-02-24 0:32 [Ocfs2-devel] report BUG: io_uring triggers umount error Heming Zhao via Ocfs2-devel 2023-02-24 6:54 ` Joseph Qi via Ocfs2-devel 2023-02-24 7:48 ` Heming Zhao via Ocfs2-devel 2023-02-24 7:52 ` Joseph Qi via Ocfs2-devel 2023-02-24 8:03 ` Heming Zhao via Ocfs2-devel 2023-02-24 10:59 ` Joseph Qi via Ocfs2-devel 2023-02-26 8:45 ` Heming Zhao via Ocfs2-devel 2023-02-26 15:22 ` Heming Zhao via Ocfs2-devel 2023-02-27 2:59 ` Joseph Qi via Ocfs2-devel 2023-02-28 4:50 ` Joseph Qi via Ocfs2-devel 2023-03-01 2:06 ` Joseph Qi via Ocfs2-devel 2023-03-01 9:49 ` Heming Zhao via Ocfs2-devel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.