[Ocfs2-devel] report BUG: io_uring triggers umount error

[Ocfs2-devel] report BUG: io_uring triggers umount error

[Heming Zhao via Ocfs2-devel]

Hello List,

I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time
he/she could fix it. This bug is blocking fstest generic/013 test case, and also
blocking fstest to do later test cases.

How to trigger:
```
git clone git://git.kernel.dk/liburing.git
cd liburing
make
cd examples
mount -t ocfs2 /dev/sda /mnt
cp /etc/hosts /mnt/a
./link-cp /mnt/a /mnt/b
umount /mnt
```

umount trigger error message:
```
# umount /mnt 
umount: /mnt: target is busy.
```

The umount error can only be triggered by liburing write operation.

Thanks,
Heming

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Joseph Qi via Ocfs2-devel]

I can reproduce this in my local VM.
I've traced ocfs2_dismount_volume and found that it hasn't been called.
So EBUSY is returned in VFS layer. I guess something wrong when doing
a copy with linked SQEs (normal copy seems no problem).

Thanks,
Joseph

On 2/24/23 8:32 AM, Heming Zhao wrote:
> Hello List,
> 
> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time
> he/she could fix it. This bug is blocking fstest generic/013 test case, and also
> blocking fstest to do later test cases.
> 
> How to trigger:
> ```
> git clone git://git.kernel.dk/liburing.git
> cd liburing
> make
> cd examples
> mount -t ocfs2 /dev/sda /mnt
> cp /etc/hosts /mnt/a
> ./link-cp /mnt/a /mnt/b
> umount /mnt
> ```
> 
> umount trigger error message:
> ```
> # umount /mnt 
> umount: /mnt: target is busy.
> ```
> 
> The umount error can only be triggered by liburing write operation.
> 
> Thanks,
> Heming

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Heming Zhao via Ocfs2-devel]

On 2/24/23 2:54 PM, Joseph Qi wrote:
> I can reproduce this in my local VM.
> I've traced ocfs2_dismount_volume and found that it hasn't been called.
> So EBUSY is returned in VFS layer. I guess something wrong when doing
> a copy with linked SQEs (normal copy seems no problem).
> 

I am inclined to agree with you. I also test liburing examples apps
on ext4 partition, everything looks fine.

I used below bpftrace method, the retval is '3'.
  bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'

It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')

Thanks,
Heming

> 
> On 2/24/23 8:32 AM, Heming Zhao wrote:
>> Hello List,
>>
>> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time
>> he/she could fix it. This bug is blocking fstest generic/013 test case, and also
>> blocking fstest to do later test cases.
>>
>> How to trigger:
>> ```
>> git clone git://git.kernel.dk/liburing.git
>> cd liburing
>> make
>> cd examples
>> mount -t ocfs2 /dev/sda /mnt
>> cp /etc/hosts /mnt/a
>> ./link-cp /mnt/a /mnt/b
>> umount /mnt
>> ```
>>
>> umount trigger error message:
>> ```
>> # umount /mnt
>> umount: /mnt: target is busy.
>> ```
>>
>> The umount error can only be triggered by liburing write operation.
>>
>> Thanks,
>> Heming


_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Joseph Qi via Ocfs2-devel]

On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
> On 2/24/23 2:54 PM, Joseph Qi wrote:
>> I can reproduce this in my local VM.
>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>> a copy with linked SQEs (normal copy seems no problem).
>>
> 
> I am inclined to agree with you. I also test liburing examples apps
> on ext4 partition, everything looks fine.
> 
> I used below bpftrace method, the retval is '3'.
>  bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
> 
> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
> 
Yes, that's the place return EBUSY.
So the problem seems to be getmnt/putmnt not match in this case.

Thanks,
Joseph

> 
>>
>> On 2/24/23 8:32 AM, Heming Zhao wrote:
>>> Hello List,
>>>
>>> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time
>>> he/she could fix it. This bug is blocking fstest generic/013 test case, and also
>>> blocking fstest to do later test cases.
>>>
>>> How to trigger:
>>> ```
>>> git clone git://git.kernel.dk/liburing.git
>>> cd liburing
>>> make
>>> cd examples
>>> mount -t ocfs2 /dev/sda /mnt
>>> cp /etc/hosts /mnt/a
>>> ./link-cp /mnt/a /mnt/b
>>> umount /mnt
>>> ```
>>>
>>> umount trigger error message:
>>> ```
>>> # umount /mnt
>>> umount: /mnt: target is busy.
>>> ```
>>>
>>> The umount error can only be triggered by liburing write operation.
>>>
>>> Thanks,
>>> Heming
> 
> 
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel@oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Heming Zhao via Ocfs2-devel]

On 2/24/23 3:52 PM, Joseph Qi wrote:
> 
> 
> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
>> On 2/24/23 2:54 PM, Joseph Qi wrote:
>>> I can reproduce this in my local VM.
>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>>> a copy with linked SQEs (normal copy seems no problem).
>>>
>>
>> I am inclined to agree with you. I also test liburing examples apps
>> on ext4 partition, everything looks fine.
>>
>> I used below bpftrace method, the retval is '3'.
>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
>>
>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
>>
> Yes, that's the place return EBUSY.
> So the problem seems to be getmnt/putmnt not match in this case.
> 

I didn't familiar with setting up kernel bi-search env. I used one last year
openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
So there is a possibility one ocfs2 commit introduced this issue.

Thanks,
Heming

> 
>>
>>>
>>> On 2/24/23 8:32 AM, Heming Zhao wrote:
>>>> Hello List,
>>>>
>>>> I found a weird bug on ocfs2. I am busying with other jobs, if anyone have time
>>>> he/she could fix it. This bug is blocking fstest generic/013 test case, and also
>>>> blocking fstest to do later test cases.
>>>>
>>>> How to trigger:
>>>> ```
>>>> git clone git://git.kernel.dk/liburing.git
>>>> cd liburing
>>>> make
>>>> cd examples
>>>> mount -t ocfs2 /dev/sda /mnt
>>>> cp /etc/hosts /mnt/a
>>>> ./link-cp /mnt/a /mnt/b
>>>> umount /mnt
>>>> ```
>>>>
>>>> umount trigger error message:
>>>> ```
>>>> # umount /mnt
>>>> umount: /mnt: target is busy.
>>>> ```
>>>>
>>>> The umount error can only be triggered by liburing write operation.
>>>>
>>>> Thanks,
>>>> Heming
>>
>>
>> _______________________________________________
>> Ocfs2-devel mailing list
>> Ocfs2-devel@oss.oracle.com
>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel


_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Joseph Qi via Ocfs2-devel]

On 2/24/23 4:03 PM, Heming Zhao wrote:
> On 2/24/23 3:52 PM, Joseph Qi wrote:
>>
>>
>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
>>> On 2/24/23 2:54 PM, Joseph Qi wrote:
>>>> I can reproduce this in my local VM.
>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>>>> a copy with linked SQEs (normal copy seems no problem).
>>>>
>>>
>>> I am inclined to agree with you. I also test liburing examples apps
>>> on ext4 partition, everything looks fine.
>>>
>>> I used below bpftrace method, the retval is '3'.
>>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
>>>
>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
>>>
>> Yes, that's the place return EBUSY.
>> So the problem seems to be getmnt/putmnt not match in this case.
>>
> 
> I didn't familiar with setting up kernel bi-search env. I used one last year
> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
> So there is a possibility one ocfs2 commit introduced this issue.
>You can checkout each mailine version like Linux 6.0, 6.1, ... and try
to check if it can be reproduced.

I've tried trace mntget/mntput using the following bpftrace script,
link-cp output shows it misses a fput.

#include <linux/mount.h>
#include <linux/string.h>

kprobe:mntget
{
	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;

	if (!strncmp(str($n), "ocfs2", 5)) {
		@[comm] += 1;

		printf("%s", kstack);
	}
}

kprobe:mntput
{
	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;

	if (!strncmp(str($n), "ocfs2", 5)) {
		@[comm] +=1;

		printf("%s", kstack);
	}
}

Joseph

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Heming Zhao via Ocfs2-devel]

On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote:
> 
> 
> On 2/24/23 4:03 PM, Heming Zhao wrote:
> > On 2/24/23 3:52 PM, Joseph Qi wrote:
> >>
> >>
> >> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
> >>> On 2/24/23 2:54 PM, Joseph Qi wrote:
> >>>> I can reproduce this in my local VM.
> >>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
> >>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
> >>>> a copy with linked SQEs (normal copy seems no problem).
> >>>>
> >>>
> >>> I am inclined to agree with you. I also test liburing examples apps
> >>> on ext4 partition, everything looks fine.
> >>>
> >>> I used below bpftrace method, the retval is '3'.
> >>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
> >>>
> >>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
> >>>
> >> Yes, that's the place return EBUSY.
> >> So the problem seems to be getmnt/putmnt not match in this case.
> >>
> > 
> > I didn't familiar with setting up kernel bi-search env. I used one last year
> > openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
> > So there is a possibility one ocfs2 commit introduced this issue.
>You can checkout each mailine version like Linux 6.0, 6.1, ... and try
> to check if it can be reproduced.

I drop this method, my machine cpu is old, compiling kernel will take too much
time.

> 
> I've tried trace mntget/mntput using the following bpftrace script,
> link-cp output shows it misses a fput.
> 
> #include <linux/mount.h>
> #include <linux/string.h>
> 
> kprobe:mntget
> {
> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
> 
> 	if (!strncmp(str($n), "ocfs2", 5)) {
> 		@[comm] += 1;
> 
> 		printf("%s", kstack);
> 	}
> }
> 
> kprobe:mntput
> {
> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
> 
> 	if (!strncmp(str($n), "ocfs2", 5)) {
> 		@[comm] +=1;
> 
> 		printf("%s", kstack);
> 	}
> }
> 

I used below script to find source file hold f_count.

e.g: ./link-cp /mnt/aa.bin /mnt/1
The script will show f_count of aa.bin only down to 3. We expect f_count value
down to 1 then trigger mntput.

```
#include <linux/mount.h>
#include <linux/string.h>
#include <linux/dcache.h>
#include <linux/path.h>
#include <linux/fs_struct.h>
#include <linux/fs.h>
#include <linux/io_uring_types.h>

k:fput
{
	$f = (struct file *)arg0;
	$cnt = $f->f_count;

	if (!strncmp(comm, "link-cp", 7)) {
		printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n",
			   	str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt));
		//printf("%s\n", kstack);
	}
}
```

- Heming

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Heming Zhao via Ocfs2-devel]

On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote:
> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote:
> >
> >
> > On 2/24/23 4:03 PM, Heming Zhao wrote:
> > > On 2/24/23 3:52 PM, Joseph Qi wrote:
> > >>
> > >>
> > >> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
> > >>> On 2/24/23 2:54 PM, Joseph Qi wrote:
> > >>>> I can reproduce this in my local VM.
> > >>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
> > >>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
> > >>>> a copy with linked SQEs (normal copy seems no problem).
> > >>>>
> > >>>
> > >>> I am inclined to agree with you. I also test liburing examples apps
> > >>> on ext4 partition, everything looks fine.
> > >>>
> > >>> I used below bpftrace method, the retval is '3'.
> > >>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
> > >>>
> > >>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
> > >>>
> > >> Yes, that's the place return EBUSY.
> > >> So the problem seems to be getmnt/putmnt not match in this case.
> > >>
> > >
> > > I didn't familiar with setting up kernel bi-search env. I used one last year
> > > openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
> > > So there is a possibility one ocfs2 commit introduced this issue.
> >You can checkout each mailine version like Linux 6.0, 6.1, ... and try
> > to check if it can be reproduced.
>
> I drop this method, my machine cpu is old, compiling kernel will take too much
> time.
>
> >
> > I've tried trace mntget/mntput using the following bpftrace script,
> > link-cp output shows it misses a fput.
> >
> > #include <linux/mount.h>
> > #include <linux/string.h>
> >
> > kprobe:mntget
> > {
> > 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
> >
> > 	if (!strncmp(str($n), "ocfs2", 5)) {
> > 		@[comm] += 1;
> >
> > 		printf("%s", kstack);
> > 	}
> > }
> >
> > kprobe:mntput
> > {
> > 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
> >
> > 	if (!strncmp(str($n), "ocfs2", 5)) {
> > 		@[comm] +=1;
> >
> > 		printf("%s", kstack);
> > 	}
> > }
> >
>
> I used below script to find source file hold f_count.
>
> e.g: ./link-cp /mnt/aa.bin /mnt/1
> The script will show f_count of aa.bin only down to 3. We expect f_count value
> down to 1 then trigger mntput.
>
> ```
> #include <linux/mount.h>
> #include <linux/string.h>
> #include <linux/dcache.h>
> #include <linux/path.h>
> #include <linux/fs_struct.h>
> #include <linux/fs.h>
> #include <linux/io_uring_types.h>
>
> k:fput
> {
> 	$f = (struct file *)arg0;
> 	$cnt = $f->f_count;
>
> 	if (!strncmp(comm, "link-cp", 7)) {
> 		printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n",
> 			   	str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt));
> 		//printf("%s\n", kstack);
> 	}
> }
> ```
>

I am not familiar with io_uring, and can't make sure my analysis is correct.
Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails
io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure
issue.

io_issue_sqe
 + io_assign_file //call fget(), but missing fput()
 + def->issue()
 |  io_read
 |   + io_iter_do_read
 |   |  ocfs2_file_read_iter
 |   |   return: -EOPNOTSUPP //iocb->ki_flags:0x8
 |   + kiocb_done
 |      + io_rw_done(&rw->kiocb, ret)
 |      |   kiocb->ki_complete(kiocb, ret)
 |      |    io_complete_rw
 |      |      __io_complete_rw_common
 |      |       + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO;
 |      |       + return true
 |      + io_req_task_queue_reissue/io_req_task_queue_fail
 |      + return IOU_ISSUE_SKIP_COMPLETE;
 |
 + return 0;

- Heming

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Joseph Qi via Ocfs2-devel]

On 2/26/23 11:22 PM, Heming Zhao wrote:
> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote:
>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote:
>>>
>>>
>>> On 2/24/23 4:03 PM, Heming Zhao wrote:
>>>> On 2/24/23 3:52 PM, Joseph Qi wrote:
>>>>>
>>>>>
>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote:
>>>>>>> I can reproduce this in my local VM.
>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>>>>>>> a copy with linked SQEs (normal copy seems no problem).
>>>>>>>
>>>>>>
>>>>>> I am inclined to agree with you. I also test liburing examples apps
>>>>>> on ext4 partition, everything looks fine.
>>>>>>
>>>>>> I used below bpftrace method, the retval is '3'.
>>>>>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
>>>>>>
>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
>>>>>>
>>>>> Yes, that's the place return EBUSY.
>>>>> So the problem seems to be getmnt/putmnt not match in this case.
>>>>>
>>>>
>>>> I didn't familiar with setting up kernel bi-search env. I used one last year
>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
>>>> So there is a possibility one ocfs2 commit introduced this issue.
>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try
>>> to check if it can be reproduced.
>>
>> I drop this method, my machine cpu is old, compiling kernel will take too much
>> time.
>>
>>>
>>> I've tried trace mntget/mntput using the following bpftrace script,
>>> link-cp output shows it misses a fput.
>>>
>>> #include <linux/mount.h>
>>> #include <linux/string.h>
>>>
>>> kprobe:mntget
>>> {
>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>
>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>> 		@[comm] += 1;
>>>
>>> 		printf("%s", kstack);
>>> 	}
>>> }
>>>
>>> kprobe:mntput
>>> {
>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>
>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>> 		@[comm] +=1;
>>>
>>> 		printf("%s", kstack);
>>> 	}
>>> }
>>>
>>
>> I used below script to find source file hold f_count.
>>
>> e.g: ./link-cp /mnt/aa.bin /mnt/1
>> The script will show f_count of aa.bin only down to 3. We expect f_count value
>> down to 1 then trigger mntput.
>>
>> ```
>> #include <linux/mount.h>
>> #include <linux/string.h>
>> #include <linux/dcache.h>
>> #include <linux/path.h>
>> #include <linux/fs_struct.h>
>> #include <linux/fs.h>
>> #include <linux/io_uring_types.h>
>>
>> k:fput
>> {
>> 	$f = (struct file *)arg0;
>> 	$cnt = $f->f_count;
>>
>> 	if (!strncmp(comm, "link-cp", 7)) {
>> 		printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n",
>> 			   	str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt));
>> 		//printf("%s\n", kstack);
>> 	}
>> }
>> ```
>>
> 
> I am not familiar with io_uring, and can't make sure my analysis is correct.
> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails
> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure
> issue.
> 
> io_issue_sqe
>  + io_assign_file //call fget(), but missing fput()
>  + def->issue()
>  |  io_read
>  |   + io_iter_do_read
>  |   |  ocfs2_file_read_iter
>  |   |   return: -EOPNOTSUPP //iocb->ki_flags:0x8
>  |   + kiocb_done
>  |      + io_rw_done(&rw->kiocb, ret)
>  |      |   kiocb->ki_complete(kiocb, ret)
>  |      |    io_complete_rw
>  |      |      __io_complete_rw_common
>  |      |       + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO;
>  |      |       + return true
>  |      + io_req_task_queue_reissue/io_req_task_queue_fail
>  |      + return IOU_ISSUE_SKIP_COMPLETE;
>  |
>  + return 0;
> 
Seems not the case.
Reissue will queue work to iowq, and fput will finally get called in
io_wq_free_work() once work is handled.

Thanks,
Joseph

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Joseph Qi via Ocfs2-devel]

On 2/27/23 10:59 AM, Joseph Qi wrote:
> 
> 
> On 2/26/23 11:22 PM, Heming Zhao wrote:
>> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote:
>>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote:
>>>>
>>>>
>>>> On 2/24/23 4:03 PM, Heming Zhao wrote:
>>>>> On 2/24/23 3:52 PM, Joseph Qi wrote:
>>>>>>
>>>>>>
>>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
>>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote:
>>>>>>>> I can reproduce this in my local VM.
>>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>>>>>>>> a copy with linked SQEs (normal copy seems no problem).
>>>>>>>>
>>>>>>>
>>>>>>> I am inclined to agree with you. I also test liburing examples apps
>>>>>>> on ext4 partition, everything looks fine.
>>>>>>>
>>>>>>> I used below bpftrace method, the retval is '3'.
>>>>>>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
>>>>>>>
>>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
>>>>>>>
>>>>>> Yes, that's the place return EBUSY.
>>>>>> So the problem seems to be getmnt/putmnt not match in this case.
>>>>>>
>>>>>
>>>>> I didn't familiar with setting up kernel bi-search env. I used one last year
>>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
>>>>> So there is a possibility one ocfs2 commit introduced this issue.
>>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try
>>>> to check if it can be reproduced.
>>>
>>> I drop this method, my machine cpu is old, compiling kernel will take too much
>>> time.
>>>
>>>>
>>>> I've tried trace mntget/mntput using the following bpftrace script,
>>>> link-cp output shows it misses a fput.
>>>>
>>>> #include <linux/mount.h>
>>>> #include <linux/string.h>
>>>>
>>>> kprobe:mntget
>>>> {
>>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>>
>>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>>> 		@[comm] += 1;
>>>>
>>>> 		printf("%s", kstack);
>>>> 	}
>>>> }
>>>>
>>>> kprobe:mntput
>>>> {
>>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>>
>>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>>> 		@[comm] +=1;
>>>>
>>>> 		printf("%s", kstack);
>>>> 	}
>>>> }
>>>>
>>>
>>> I used below script to find source file hold f_count.
>>>
>>> e.g: ./link-cp /mnt/aa.bin /mnt/1
>>> The script will show f_count of aa.bin only down to 3. We expect f_count value
>>> down to 1 then trigger mntput.
>>>
>>> ```
>>> #include <linux/mount.h>
>>> #include <linux/string.h>
>>> #include <linux/dcache.h>
>>> #include <linux/path.h>
>>> #include <linux/fs_struct.h>
>>> #include <linux/fs.h>
>>> #include <linux/io_uring_types.h>
>>>
>>> k:fput
>>> {
>>> 	$f = (struct file *)arg0;
>>> 	$cnt = $f->f_count;
>>>
>>> 	if (!strncmp(comm, "link-cp", 7)) {
>>> 		printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n",
>>> 			   	str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt));
>>> 		//printf("%s\n", kstack);
>>> 	}
>>> }
>>> ```
>>>
>>
>> I am not familiar with io_uring, and can't make sure my analysis is correct.
>> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails
>> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure
>> issue.
>>
>> io_issue_sqe
>>  + io_assign_file //call fget(), but missing fput()
>>  + def->issue()
>>  |  io_read
>>  |   + io_iter_do_read
>>  |   |  ocfs2_file_read_iter
>>  |   |   return: -EOPNOTSUPP //iocb->ki_flags:0x8
>>  |   + kiocb_done
>>  |      + io_rw_done(&rw->kiocb, ret)
>>  |      |   kiocb->ki_complete(kiocb, ret)
>>  |      |    io_complete_rw
>>  |      |      __io_complete_rw_common
>>  |      |       + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO;
>>  |      |       + return true
>>  |      + io_req_task_queue_reissue/io_req_task_queue_fail
>>  |      + return IOU_ISSUE_SKIP_COMPLETE;
>>  |
>>  + return 0;
>>
> Seems not the case.
> Reissue will queue work to iowq, and fput will finally get called in
> io_wq_free_work() once work is handled.
> I think I've found the root cause, will send a patch later.

Thanks,
Joseph

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Joseph Qi via Ocfs2-devel]

Hi,

On 2/27/23 10:59 AM, Joseph Qi via Ocfs2-devel wrote:
> 
> 
> On 2/26/23 11:22 PM, Heming Zhao wrote:
>> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote:
>>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote:
>>>>
>>>>
>>>> On 2/24/23 4:03 PM, Heming Zhao wrote:
>>>>> On 2/24/23 3:52 PM, Joseph Qi wrote:
>>>>>>
>>>>>>
>>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
>>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote:
>>>>>>>> I can reproduce this in my local VM.
>>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>>>>>>>> a copy with linked SQEs (normal copy seems no problem).
>>>>>>>>
>>>>>>>
>>>>>>> I am inclined to agree with you. I also test liburing examples apps
>>>>>>> on ext4 partition, everything looks fine.
>>>>>>>
>>>>>>> I used below bpftrace method, the retval is '3'.
>>>>>>>   bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
>>>>>>>
>>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
>>>>>>>
>>>>>> Yes, that's the place return EBUSY.
>>>>>> So the problem seems to be getmnt/putmnt not match in this case.
>>>>>>
>>>>>
>>>>> I didn't familiar with setting up kernel bi-search env. I used one last year
>>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
>>>>> So there is a possibility one ocfs2 commit introduced this issue.
>>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try
>>>> to check if it can be reproduced.
>>>
>>> I drop this method, my machine cpu is old, compiling kernel will take too much
>>> time.
>>>
>>>>
>>>> I've tried trace mntget/mntput using the following bpftrace script,
>>>> link-cp output shows it misses a fput.
>>>>
>>>> #include <linux/mount.h>
>>>> #include <linux/string.h>
>>>>
>>>> kprobe:mntget
>>>> {
>>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>>
>>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>>> 		@[comm] += 1;
>>>>
>>>> 		printf("%s", kstack);
>>>> 	}
>>>> }
>>>>
>>>> kprobe:mntput
>>>> {
>>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>>
>>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>>> 		@[comm] +=1;
>>>>
>>>> 		printf("%s", kstack);
>>>> 	}
>>>> }
>>>>
>>>
>>> I used below script to find source file hold f_count.
>>>
>>> e.g: ./link-cp /mnt/aa.bin /mnt/1
>>> The script will show f_count of aa.bin only down to 3. We expect f_count value
>>> down to 1 then trigger mntput.
>>>
>>> ```
>>> #include <linux/mount.h>
>>> #include <linux/string.h>
>>> #include <linux/dcache.h>
>>> #include <linux/path.h>
>>> #include <linux/fs_struct.h>
>>> #include <linux/fs.h>
>>> #include <linux/io_uring_types.h>
>>>
>>> k:fput
>>> {
>>> 	$f = (struct file *)arg0;
>>> 	$cnt = $f->f_count;
>>>
>>> 	if (!strncmp(comm, "link-cp", 7)) {
>>> 		printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n",
>>> 			   	str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt));
>>> 		//printf("%s\n", kstack);
>>> 	}
>>> }
>>> ```
>>>
>>
>> I am not familiar with io_uring, and can't make sure my analysis is correct.
>> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails
>> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure
>> issue.
>>
>> io_issue_sqe
>>  + io_assign_file //call fget(), but missing fput()
>>  + def->issue()
>>  |  io_read
>>  |   + io_iter_do_read
>>  |   |  ocfs2_file_read_iter
>>  |   |   return: -EOPNOTSUPP //iocb->ki_flags:0x8
>>  |   + kiocb_done
>>  |      + io_rw_done(&rw->kiocb, ret)
>>  |      |   kiocb->ki_complete(kiocb, ret)
>>  |      |    io_complete_rw
>>  |      |      __io_complete_rw_common
>>  |      |       + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO;
>>  |      |       + return true
>>  |      + io_req_task_queue_reissue/io_req_task_queue_fail
>>  |      + return IOU_ISSUE_SKIP_COMPLETE;
>>  |
>>  + return 0;
>>
> Seems not the case.
> Reissue will queue work to iowq, and fput will finally get called in
> io_wq_free_work() once work is handled.
> 
I've sent a fix to io_uring, please refer:
https://lore.kernel.org/io-uring/167758917178.12826.13481934362166793957.b4-ty@kernel.dk/T/#t

Thanks,
Joseph

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Re: [Ocfs2-devel] report BUG: io_uring triggers umount error

[Heming Zhao via Ocfs2-devel]

On 3/1/23 10:06 AM, Joseph Qi wrote:
> Hi,
> 
> On 2/27/23 10:59 AM, Joseph Qi via Ocfs2-devel wrote:
>>
>>
>> On 2/26/23 11:22 PM, Heming Zhao wrote:
>>> On Sun, Feb 26, 2023 at 04:45:44PM +0800, Heming Zhao via Ocfs2-devel wrote:
>>>> On Fri, Feb 24, 2023 at 06:59:47PM +0800, Joseph Qi wrote:
>>>>>
>>>>>
>>>>> On 2/24/23 4:03 PM, Heming Zhao wrote:
>>>>>> On 2/24/23 3:52 PM, Joseph Qi wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2/24/23 3:48 PM, Heming Zhao via Ocfs2-devel wrote:
>>>>>>>> On 2/24/23 2:54 PM, Joseph Qi wrote:
>>>>>>>>> I can reproduce this in my local VM.
>>>>>>>>> I've traced ocfs2_dismount_volume and found that it hasn't been called.
>>>>>>>>> So EBUSY is returned in VFS layer. I guess something wrong when doing
>>>>>>>>> a copy with linked SQEs (normal copy seems no problem).
>>>>>>>>>
>>>>>>>>
>>>>>>>> I am inclined to agree with you. I also test liburing examples apps
>>>>>>>> on ext4 partition, everything looks fine.
>>>>>>>>
>>>>>>>> I used below bpftrace method, the retval is '3'.
>>>>>>>>    bpftrace -e 'kr:mnt_get_count{printf("%d\n", retval);}'
>>>>>>>>
>>>>>>>> It responds to flow: path_umount() => do_umount => mnt_get_count (gets '3')
>>>>>>>>
>>>>>>> Yes, that's the place return EBUSY.
>>>>>>> So the problem seems to be getmnt/putmnt not match in this case.
>>>>>>>
>>>>>>
>>>>>> I didn't familiar with setting up kernel bi-search env. I used one last year
>>>>>> openSUSE tumblweed (with kernel 5.16.2), this umount issue doesn't exist.
>>>>>> So there is a possibility one ocfs2 commit introduced this issue.
>>>>> You can checkout each mailine version like Linux 6.0, 6.1, ... and try
>>>>> to check if it can be reproduced.
>>>>
>>>> I drop this method, my machine cpu is old, compiling kernel will take too much
>>>> time.
>>>>
>>>>>
>>>>> I've tried trace mntget/mntput using the following bpftrace script,
>>>>> link-cp output shows it misses a fput.
>>>>>
>>>>> #include <linux/mount.h>
>>>>> #include <linux/string.h>
>>>>>
>>>>> kprobe:mntget
>>>>> {
>>>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>>>
>>>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>>>> 		@[comm] += 1;
>>>>>
>>>>> 		printf("%s", kstack);
>>>>> 	}
>>>>> }
>>>>>
>>>>> kprobe:mntput
>>>>> {
>>>>> 	$n = ((struct vfsmount *)arg0)->mnt_sb->s_type->name;
>>>>>
>>>>> 	if (!strncmp(str($n), "ocfs2", 5)) {
>>>>> 		@[comm] +=1;
>>>>>
>>>>> 		printf("%s", kstack);
>>>>> 	}
>>>>> }
>>>>>
>>>>
>>>> I used below script to find source file hold f_count.
>>>>
>>>> e.g: ./link-cp /mnt/aa.bin /mnt/1
>>>> The script will show f_count of aa.bin only down to 3. We expect f_count value
>>>> down to 1 then trigger mntput.
>>>>
>>>> ```
>>>> #include <linux/mount.h>
>>>> #include <linux/string.h>
>>>> #include <linux/dcache.h>
>>>> #include <linux/path.h>
>>>> #include <linux/fs_struct.h>
>>>> #include <linux/fs.h>
>>>> #include <linux/io_uring_types.h>
>>>>
>>>> k:fput
>>>> {
>>>> 	$f = (struct file *)arg0;
>>>> 	$cnt = $f->f_count;
>>>>
>>>> 	if (!strncmp(comm, "link-cp", 7)) {
>>>> 		printf("\n [link-cp] fput() name:%s f_count:%lld <%p> mnt:%p\n",
>>>> 			   	str($f->f_path.dentry->d_name.name), $cnt.counter, kptr($f), kptr($f->f_path.mnt));
>>>> 		//printf("%s\n", kstack);
>>>> 	}
>>>> }
>>>> ```
>>>>
>>>
>>> I am not familiar with io_uring, and can't make sure my analysis is correct.
>>> Because ocfs2_file_read_iter() returns -EOPNOTSUPP, then fails
>>> io_iter_do_read(), the io_issue_sqe missing fput() which causes umount failure
>>> issue.
>>>
>>> io_issue_sqe
>>>   + io_assign_file //call fget(), but missing fput()
>>>   + def->issue()
>>>   |  io_read
>>>   |   + io_iter_do_read
>>>   |   |  ocfs2_file_read_iter
>>>   |   |   return: -EOPNOTSUPP //iocb->ki_flags:0x8
>>>   |   + kiocb_done
>>>   |      + io_rw_done(&rw->kiocb, ret)
>>>   |      |   kiocb->ki_complete(kiocb, ret)
>>>   |      |    io_complete_rw
>>>   |      |      __io_complete_rw_common
>>>   |      |       + req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO;
>>>   |      |       + return true
>>>   |      + io_req_task_queue_reissue/io_req_task_queue_fail
>>>   |      + return IOU_ISSUE_SKIP_COMPLETE;
>>>   |
>>>   + return 0;
>>>
>> Seems not the case.
>> Reissue will queue work to iowq, and fput will finally get called in
>> io_wq_free_work() once work is handled.
>>
> I've sent a fix to io_uring, please refer:
> https://lore.kernel.org/io-uring/167758917178.12826.13481934362166793957.b4-ty@kernel.dk/T/#t
> 
> Thanks,
> Joseph
> 

Got it. Thank you for your quickly fix.

Let's make ocfs2 more powerful/stable.



- Heming

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel