Reading from fuse pipe fails with EBADFD

Reading from fuse pipe fails with EBADFD

[Nikolaus Rath]

Hi,

When testing FUSE under heavy load, I am occasionally getting EBADFD
errors when reading from the fuse pipe.

Does anyone have an idea what might cause this, and how to debug it
further?

Unfortunately I can't tell if this happens upon read() or upon
splice(). I've extended the code now, so the next time it happens I will
be able to tell.

I am testing with kernel 4.15 from Ubuntu Bionic.

So far this has happened twice in about 3 hours of testing. For the
testing, I am repeatedly running the same 3 workloads, without
remounting. Each workload takes ~ 15-30 seconds.

Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Reading from fuse pipe fails with EBADFD

[Nikolaus Rath]

Hi,

Really no one any suggestion for debugging this?

(Adding some more people who recently worked on fs/fuse)

Best,
-Nikolaus

On Nov 27 2018, Nikolaus Rath <Nikolaus@rath.org> wrote:
> Hi,
>
> When testing FUSE under heavy load, I am occasionally getting EBADFD
> errors when reading from the fuse pipe.
>
> Does anyone have an idea what might cause this, and how to debug it
> further?
>
> Unfortunately I can't tell if this happens upon read() or upon
> splice(). I've extended the code now, so the next time it happens I will
> be able to tell.
>
> I am testing with kernel 4.15 from Ubuntu Bionic.
>
> So far this has happened twice in about 3 hours of testing. For the
> testing, I am repeatedly running the same 3 workloads, without
> remounting. Each workload takes ~ 15-30 seconds.
>
> Best,
> -Nikolaus
>
> -- 
> GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
>              »Time flies like an arrow, fruit flies like a Banana.«


-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Reading from fuse pipe fails with EBADFD

[Miklos Szeredi]

On Tue, Dec 4, 2018 at 10:03 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> Hi,
>
> Really no one any suggestion for debugging this?
>
> (Adding some more people who recently worked on fs/fuse)
>
> Best,
> -Nikolaus
>
> On Nov 27 2018, Nikolaus Rath <Nikolaus@rath.org> wrote:
> > Hi,
> >
> > When testing FUSE under heavy load, I am occasionally getting EBADFD
> > errors when reading from the fuse pipe.

EBADFD or EBADF?

Thanks,
Miklos

Re: [fuse-devel] Reading from fuse pipe fails with EBADFD

[Nikolaus Rath]

On Dec 04 2018, Miklos Szeredi <mszeredi@redhat.com> wrote:
> On Tue, Dec 4, 2018 at 10:03 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
>>
>> Hi,
>>
>> Really no one any suggestion for debugging this?
>>
>> (Adding some more people who recently worked on fs/fuse)
>>
>> Best,
>> -Nikolaus
>>
>> On Nov 27 2018, Nikolaus Rath <Nikolaus@rath.org> wrote:
>> > Hi,
>> >
>> > When testing FUSE under heavy load, I am occasionally getting EBADFD
>> > errors when reading from the fuse pipe.
>
> EBADFD or EBADF?

EBADF ("Bad file descriptor"), sorry.


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] Reading from fuse pipe fails with EBADFD

[Miklos Szeredi]

On Tue, Dec 4, 2018 at 8:02 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> On Dec 04 2018, Miklos Szeredi <mszeredi@redhat.com> wrote:
> > On Tue, Dec 4, 2018 at 10:03 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
> >>
> >> Hi,
> >>
> >> Really no one any suggestion for debugging this?
> >>
> >> (Adding some more people who recently worked on fs/fuse)
> >>
> >> Best,
> >> -Nikolaus
> >>
> >> On Nov 27 2018, Nikolaus Rath <Nikolaus@rath.org> wrote:
> >> > Hi,
> >> >
> >> > When testing FUSE under heavy load, I am occasionally getting EBADFD
> >> > errors when reading from the fuse pipe.
> >
> > EBADFD or EBADF?
>
> EBADF ("Bad file descriptor"), sorry.

Can you run the thing with "strace -f  ..."?

Thanks,
Miklos

Re: [fuse-devel] Reading from fuse pipe fails with EBADFD

[Nikolaus Rath]

On Dec 10 2018, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Tue, Dec 4, 2018 at 8:02 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>>
>> On Dec 04 2018, Miklos Szeredi <mszeredi@redhat.com> wrote:
>> > On Tue, Dec 4, 2018 at 10:03 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Really no one any suggestion for debugging this?
>> >>
>> >> (Adding some more people who recently worked on fs/fuse)
>> >>
>> >> Best,
>> >> -Nikolaus
>> >>
>> >> On Nov 27 2018, Nikolaus Rath <Nikolaus@rath.org> wrote:
>> >> > Hi,
>> >> >
>> >> > When testing FUSE under heavy load, I am occasionally getting EBADFD
>> >> > errors when reading from the fuse pipe.
>> >
>> > EBADFD or EBADF?
>>
>> EBADF ("Bad file descriptor"), sorry.
>
> Can you run the thing with "strace -f  ..."?

Apologies for the delayed response. I have been trying to reproduce this
but have instead run into another problem: the *client* getting spurious
EBADF warnings. I am not sure if this is related or unrelated, and it
has been hard to debug because it does not happen under strace:

$ find mnt > /dev/null
find: ‘mnt/modules/4.18.0-0.bpo.3-amd64/kernel/net/8021q’: Bad file descriptor
find: ‘mnt/modules/4.18.0-0.bpo.3-amd64/kernel/drivers/net/ethernet/natsemi’: Bad file descriptor

This happens in roughly 1 in 2 runs, and the affected directory entries
are always different. Disabling readdirplus does not make a difference.

$ strace -o log find mnt > /dev/null

never gave in error in roughly 20 attempts.

Similarly, enabling fuse debug logging also makes the problem go away.

I also found some odd warnings in dmesg:

[24472.435256] fuse: trying to steal weird page
[24472.435261]   page=00000000b5b89670 index=0 flags=17fffc0000000ad, count=1, mapcount=0, map

..happens a lot (and I just wrote another email about it), and 

[24473.170110] VFS: Lookup of '4.18.0-0.bpo.1-amd64' in fuse fuse would have caused loop

happened only a few times (and the `fuse` string is indeed
duplicated). When this happens, `find` complains that:

$ find mnt > /dev/null
find: File system loop detected; ‘mnt/modules/4.18.0-0.bpo.1-amd64/kernel/drivers/w1’ is part of the same file system loop as ‘mnt/modules’


Does this ring any bells with anyone?


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] Reading from fuse pipe fails with EBADFD

[Nikolaus Rath]

On Dec 26 2018, Nikolaus Rath <Nikolaus@rath.org> wrote:
>>> >> > When testing FUSE under heavy load, I am occasionally getting EBADFD
>>> >> > errors when reading from the fuse pipe.
>>> >
>>> > EBADFD or EBADF?
>>>
>>> EBADF ("Bad file descriptor"), sorry.
>>
>> Can you run the thing with "strace -f  ..."?
>
> Apologies for the delayed response. I have been trying to reproduce this
> but have instead run into another problem: the *client* getting spurious
> EBADF warnings. I am not sure if this is related or unrelated, and it
> has been hard to debug because it does not happen under strace:
[..]

I believe I have figured it out. I was mistakenly assuming that the bad
file descriptor was the fuse pipe - but it was actually the target file
descriptor (I managed to completely forget that splice works on two file
descriptors).

The root cause was mistakenly closing a file descriptor twice and not
checking the return value. Because of the missing return value check,
the error went unnoticed almost all the time - except when a different
thread managed to re-use the fd for different purposes between the first
and second close.

Apologies for taking everyone's time for what was actually a filesystem
bug.


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«