Interrupt on readdirplus?

Interrupt on readdirplus?

[Han-Wen Nienhuys]

Hi there,

I have noticed Go-FUSE test failures of late, that seem to originate
in (changed?) kernel behavior. The problems looks like this:

12:54:13.385435 rx 20: OPENDIR n1  p330882
12:54:13.385514 tx 20:     OK, {Fh 1 }
12:54:13.385838 rx 22: READDIRPLUS n1 {Fh 1 [0 +4096)  L 0 LARGEFILE}  p330882
12:54:13.385844 rx 23: INTERRUPT n0 {ix 22}  p0
12:54:13.386114 tx 22:     OK,  4000b data "\x02\x00\x00\x00\x00\x00\x00\x00"...
12:54:13.386642 rx 24: READDIRPLUS n1 {Fh 1 [1 +4096)  L 0 LARGEFILE}  p330882
12:54:13.386849 tx 24:     95=operation not supported

As you can see, the kernel attempts to interrupt the READDIRPLUS
operation, but go-fuse ignores the interrupt and returns 25 entries.
The kernel somehow thinks that only 1 entry was consumed, and issues
the next READDIRPLUS at offset 1. If go-fuse ignores the faulty offset
and continues the listing (ie. continuing with entry 25), the test
passes.

Is this behavior of the kernel expected or a bug?

I am redoing the API for directory listing to support cacheable and
seekable directories, and in the new version, this looks like a
directory seek. If the file system does not support seekable
directories, I must return some kind of error (which is the ENOTSUP
you can see in the log above).

I started seeing this after upgrading to Fedora 40. My kernel is
6.10.7-200.fc40.x86_64

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

Hi Han-Wen,

On 9/10/24 14:45, Han-Wen Nienhuys wrote:
> Hi there,
> 
> I have noticed Go-FUSE test failures of late, that seem to originate
> in (changed?) kernel behavior. The problems looks like this:
> 
> 12:54:13.385435 rx 20: OPENDIR n1  p330882
> 12:54:13.385514 tx 20:     OK, {Fh 1 }
> 12:54:13.385838 rx 22: READDIRPLUS n1 {Fh 1 [0 +4096)  L 0 LARGEFILE}  p330882
> 12:54:13.385844 rx 23: INTERRUPT n0 {ix 22}  p0
> 12:54:13.386114 tx 22:     OK,  4000b data "\x02\x00\x00\x00\x00\x00\x00\x00"...
> 12:54:13.386642 rx 24: READDIRPLUS n1 {Fh 1 [1 +4096)  L 0 LARGEFILE}  p330882
> 12:54:13.386849 tx 24:     95=operation not supported
> 
> As you can see, the kernel attempts to interrupt the READDIRPLUS

do you where the interrupt comes from? Is your test interrupting
interrupting readdir?

> operation, but go-fuse ignores the interrupt and returns 25 entries.
> The kernel somehow thinks that only 1 entry was consumed, and issues
> the next READDIRPLUS at offset 1. If go-fuse ignores the faulty offset
> and continues the listing (ie. continuing with entry 25), the test
> passes.
> 
> Is this behavior of the kernel expected or a bug?
> 
> I am redoing the API for directory listing to support cacheable and
> seekable directories, and in the new version, this looks like a
> directory seek. If the file system does not support seekable
> directories, I must return some kind of error (which is the ENOTSUP
> you can see in the log above).

Is this with or without FOPEN_CACHE_DIR? Would be helpful to know
if FOPEN_CACHE_DIR - fuse kernel code is quite different when this
is set.

> 
> I started seeing this after upgrading to Fedora 40. My kernel is
> 6.10.7-200.fc40.x86_64
> 

Would be interesting to know your kernel version before? There is
commit cdf6ac2a03d2, which removes a readdir lock. Although the
commit message explains why it is not needed anymore.


Thanks,
Bernd

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Tue, Sep 10, 2024 at 5:58 PM Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:
> > I have noticed Go-FUSE test failures of late, that seem to originate
> > in (changed?) kernel behavior. The problems looks like this:
> >
> > 12:54:13.385435 rx 20: OPENDIR n1  p330882
> > 12:54:13.385514 tx 20:     OK, {Fh 1 }
> > 12:54:13.385838 rx 22: READDIRPLUS n1 {Fh 1 [0 +4096)  L 0 LARGEFILE}  p330882
> > 12:54:13.385844 rx 23: INTERRUPT n0 {ix 22}  p0
> > 12:54:13.386114 tx 22:     OK,  4000b data "\x02\x00\x00\x00\x00\x00\x00\x00"...
> > 12:54:13.386642 rx 24: READDIRPLUS n1 {Fh 1 [1 +4096)  L 0 LARGEFILE}  p330882
> > 12:54:13.386849 tx 24:     95=operation not supported
> >
> > As you can see, the kernel attempts to interrupt the READDIRPLUS
>
> do you where the interrupt comes from? Is your test interrupting
> interrupting readdir?

I did not write code to issue interrupts, but it is possible that the
Go runtime does something behind my back. The debug output lists "p0";
is there a reason that the INTERRUPT opcode does not provide an
originating PID ? How would I discover who or what is generating the
interrupts? Will they show up if I run strace on the test binary?

I straced the test binary, below is a section where an interrupt
happens just before a directory seek. Could tgkill(SIGURG) cause an
interrupt? It happens just before the READDIRPLUS op (',') and the
INTERRUPT operation ('$') are read.

[pid 371933] writev(10,
[{iov_base="\260\17\0\0\0\0\0\0\20\0\0\0\0\0\0\0", iov_len=16},
{iov_base="\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\
0\0\0"..., iov_len=4000}], 2 <unfinished ...>
[pid 371931] <... nanosleep resumed>NULL) = 0
[pid 371933] <... writev resumed>)      = 4016
[pid 371931] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 371933] read(10,  <unfinished ...>
[pid 371934] <... getdents64 resumed>0xc0002ee000 /* 25 entries */, 8192) = 800
[pid 371931] <... nanosleep resumed>NULL) = 0
[pid 371934] futex(0xc000059148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 371931] getpid( <unfinished ...>
[pid 371934] <... futex resumed>)       = 1
[pid 371932] <... futex resumed>)       = 0
[pid 371931] <... getpid resumed>)      = 371930
[pid 371934] getdents64(7,  <unfinished ...>
[pid 371932] futex(0xc000059148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 371931] tgkill(371930, 371934, SIGURG <unfinished ...>
[pid 371935] <... read
resumed>"P\0\0\0,\0\0\0\22\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
131200) = 80
    # , = readdirplus
[pid 371933] <... read
resumed>"0\0\0\0$\0\0\0\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
131200) = 48
    # $ = interrupt.
[pid 371931] <... tgkill resumed>)      = 0
[pid 371933] futex(0xc000059148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 371931] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 371933] <... futex resumed>)       = 1
[pid 371932] <... futex resumed>)       = 0
[pid 371933] write(2, "19:24:40.813294 doInterrupt\n", 28 <unfinished ...>
19:24:40.813294 doInterrupt

A bit of browsing through the Go source code suggests that SIGURG is
used to preempt long-running goroutines, so it could be issued more or
less at random.

Nevertheless, FUSE should also not be reissuing the reads, even if
there were interrupts, right?

> > operation, but go-fuse ignores the interrupt and returns 25 entries.
> > The kernel somehow thinks that only 1 entry was consumed, and issues
> > the next READDIRPLUS at offset 1. If go-fuse ignores the faulty offset
> > and continues the listing (ie. continuing with entry 25), the test
> > passes.
> >
> > Is this behavior of the kernel expected or a bug?
> >
> > I am redoing the API for directory listing to support cacheable and
> > seekable directories, and in the new version, this looks like a
> > directory seek. If the file system does not support seekable
> > directories, I must return some kind of error (which is the ENOTSUP
> > you can see in the log above).
>
> Is this with or without FOPEN_CACHE_DIR? Would be helpful to know
> if FOPEN_CACHE_DIR - fuse kernel code is quite different when this
> is set.

without FOPEN_CACHE_DIR.

> > I started seeing this after upgrading to Fedora 40. My kernel is
> > 6.10.7-200.fc40.x86_64
> >
>
> Would be interesting to know your kernel version before? There is
> commit cdf6ac2a03d2, which removes a readdir lock. Although the
> commit message explains why it is not needed anymore.

The failure happens in single threaded loads, so a race condition
seems unlikely.

Let me try with other kernel versions.

Also, I realize that Fedora 40 also upgraded the Go compiler, and may
have made SIGURG be triggered more frequently?

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

On 9/10/24 19:38, Han-Wen Nienhuys wrote:
> On Tue, Sep 10, 2024 at 5:58 PM Bernd Schubert
> <bernd.schubert@fastmail.fm> wrote:
>>> I have noticed Go-FUSE test failures of late, that seem to originate
>>> in (changed?) kernel behavior. The problems looks like this:
>>>
>>> 12:54:13.385435 rx 20: OPENDIR n1  p330882
>>> 12:54:13.385514 tx 20:     OK, {Fh 1 }
>>> 12:54:13.385838 rx 22: READDIRPLUS n1 {Fh 1 [0 +4096)  L 0 LARGEFILE}  p330882
>>> 12:54:13.385844 rx 23: INTERRUPT n0 {ix 22}  p0
>>> 12:54:13.386114 tx 22:     OK,  4000b data "\x02\x00\x00\x00\x00\x00\x00\x00"...
>>> 12:54:13.386642 rx 24: READDIRPLUS n1 {Fh 1 [1 +4096)  L 0 LARGEFILE}  p330882
>>> 12:54:13.386849 tx 24:     95=operation not supported
>>>
>>> As you can see, the kernel attempts to interrupt the READDIRPLUS
>>
>> do you where the interrupt comes from? Is your test interrupting
>> interrupting readdir?
> 
> I did not write code to issue interrupts, but it is possible that the
> Go runtime does something behind my back. The debug output lists "p0";
> is there a reason that the INTERRUPT opcode does not provide an
> originating PID ? How would I discover who or what is generating the
> interrupts? Will they show up if I run strace on the test binary?
> 
> I straced the test binary, below is a section where an interrupt
> happens just before a directory seek. Could tgkill(SIGURG) cause an
> interrupt? It happens just before the READDIRPLUS op (',') and the
> INTERRUPT operation ('$') are read.
> 
> [pid 371933] writev(10,
> [{iov_base="\260\17\0\0\0\0\0\0\20\0\0\0\0\0\0\0", iov_len=16},
> {iov_base="\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\
> 0\0\0"..., iov_len=4000}], 2 <unfinished ...>
> [pid 371931] <... nanosleep resumed>NULL) = 0
> [pid 371933] <... writev resumed>)      = 4016
> [pid 371931] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
> [pid 371933] read(10,  <unfinished ...>
> [pid 371934] <... getdents64 resumed>0xc0002ee000 /* 25 entries */, 8192) = 800
> [pid 371931] <... nanosleep resumed>NULL) = 0
> [pid 371934] futex(0xc000059148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 371931] getpid( <unfinished ...>
> [pid 371934] <... futex resumed>)       = 1
> [pid 371932] <... futex resumed>)       = 0
> [pid 371931] <... getpid resumed>)      = 371930
> [pid 371934] getdents64(7,  <unfinished ...>
> [pid 371932] futex(0xc000059148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
> [pid 371931] tgkill(371930, 371934, SIGURG <unfinished ...>
> [pid 371935] <... read
> resumed>"P\0\0\0,\0\0\0\22\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 131200) = 80
>     # , = readdirplus
> [pid 371933] <... read
> resumed>"0\0\0\0$\0\0\0\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 131200) = 48
>     # $ = interrupt.
> [pid 371931] <... tgkill resumed>)      = 0
> [pid 371933] futex(0xc000059148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 371931] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
> [pid 371933] <... futex resumed>)       = 1
> [pid 371932] <... futex resumed>)       = 0
> [pid 371933] write(2, "19:24:40.813294 doInterrupt\n", 28 <unfinished ...>
> 19:24:40.813294 doInterrupt
> 
> A bit of browsing through the Go source code suggests that SIGURG is
> used to preempt long-running goroutines, so it could be issued more or
> less at random.
> 
> Nevertheless, FUSE should also not be reissuing the reads, even if
> there were interrupts, right?
> 
>>> operation, but go-fuse ignores the interrupt and returns 25 entries.
>>> The kernel somehow thinks that only 1 entry was consumed, and issues
>>> the next READDIRPLUS at offset 1. If go-fuse ignores the faulty offset
>>> and continues the listing (ie. continuing with entry 25), the test
>>> passes.
>>>
>>> Is this behavior of the kernel expected or a bug?
>>>
>>> I am redoing the API for directory listing to support cacheable and
>>> seekable directories, and in the new version, this looks like a
>>> directory seek. If the file system does not support seekable
>>> directories, I must return some kind of error (which is the ENOTSUP
>>> you can see in the log above).
>>
>> Is this with or without FOPEN_CACHE_DIR? Would be helpful to know
>> if FOPEN_CACHE_DIR - fuse kernel code is quite different when this
>> is set.
> 
> without FOPEN_CACHE_DIR.

So we should down to this function

https://elixir.bootlin.com/linux/v6.10.9/source/fs/fuse/readdir.c#L286

I still don't understand what is actually the failure in the test, does 
it show entries multiple times (I wouldn't understand how that could 
happen)? From the code, I would expect that you get missing entries if
you ignore the offset.

I'm not sure I ever looked into filldir64 before, but I find it
interesting that a pending signal is only checked when prev_reclen is
set.

https://elixir.bootlin.com/linux/v6.10.9/source/fs/readdir.c#L399

I.e. it accepts the first entry and then fails? parse_dirplusfile()
has to go over the entire list in the buffer, as you have a lookup
count on it. But it shouldn't list these entries - which is why
I'm confused what is the actual issue you are seeing

> 
>>> I started seeing this after upgrading to Fedora 40. My kernel is
>>> 6.10.7-200.fc40.x86_64
>>>
>>
>> Would be interesting to know your kernel version before? There is
>> commit cdf6ac2a03d2, which removes a readdir lock. Although the
>> commit message explains why it is not needed anymore.
> 
> The failure happens in single threaded loads, so a race condition
> seems unlikely.
> 
> Let me try with other kernel versions.
> 
> Also, I realize that Fedora 40 also upgraded the Go compiler, and may
> have made SIGURG be triggered more frequently?
> 


Thanks,
Bernd

Re: Interrupt on readdirplus?

[Joanne Koong]

On Tue, Sep 10, 2024 at 10:44 AM Han-Wen Nienhuys <hanwenn@gmail.com> wrote:
>
> On Tue, Sep 10, 2024 at 5:58 PM Bernd Schubert
> <bernd.schubert@fastmail.fm> wrote:
> > > I have noticed Go-FUSE test failures of late, that seem to originate
> > > in (changed?) kernel behavior. The problems looks like this:
> > >
> > > 12:54:13.385435 rx 20: OPENDIR n1  p330882
> > > 12:54:13.385514 tx 20:     OK, {Fh 1 }
> > > 12:54:13.385838 rx 22: READDIRPLUS n1 {Fh 1 [0 +4096)  L 0 LARGEFILE}  p330882
> > > 12:54:13.385844 rx 23: INTERRUPT n0 {ix 22}  p0
> > > 12:54:13.386114 tx 22:     OK,  4000b data "\x02\x00\x00\x00\x00\x00\x00\x00"...
> > > 12:54:13.386642 rx 24: READDIRPLUS n1 {Fh 1 [1 +4096)  L 0 LARGEFILE}  p330882
> > > 12:54:13.386849 tx 24:     95=operation not supported
> > >
> > > As you can see, the kernel attempts to interrupt the READDIRPLUS
> >
> > do you where the interrupt comes from? Is your test interrupting
> > interrupting readdir?
>
> I did not write code to issue interrupts, but it is possible that the
> Go runtime does something behind my back. The debug output lists "p0";
> is there a reason that the INTERRUPT opcode does not provide an
> originating PID ? How would I discover who or what is generating the
> interrupts? Will they show up if I run strace on the test binary?
>
> I straced the test binary, below is a section where an interrupt
> happens just before a directory seek. Could tgkill(SIGURG) cause an
> interrupt? It happens just before the READDIRPLUS op (',') and the
> INTERRUPT operation ('$') are read.
>
> [pid 371933] writev(10,
> [{iov_base="\260\17\0\0\0\0\0\0\20\0\0\0\0\0\0\0", iov_len=16},
> {iov_base="\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\
> 0\0\0"..., iov_len=4000}], 2 <unfinished ...>
> [pid 371931] <... nanosleep resumed>NULL) = 0
> [pid 371933] <... writev resumed>)      = 4016
> [pid 371931] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
> [pid 371933] read(10,  <unfinished ...>
> [pid 371934] <... getdents64 resumed>0xc0002ee000 /* 25 entries */, 8192) = 800
> [pid 371931] <... nanosleep resumed>NULL) = 0
> [pid 371934] futex(0xc000059148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 371931] getpid( <unfinished ...>
> [pid 371934] <... futex resumed>)       = 1
> [pid 371932] <... futex resumed>)       = 0
> [pid 371931] <... getpid resumed>)      = 371930
> [pid 371934] getdents64(7,  <unfinished ...>
> [pid 371932] futex(0xc000059148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
> [pid 371931] tgkill(371930, 371934, SIGURG <unfinished ...>
> [pid 371935] <... read
> resumed>"P\0\0\0,\0\0\0\22\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 131200) = 80
>     # , = readdirplus
> [pid 371933] <... read
> resumed>"0\0\0\0$\0\0\0\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 131200) = 48
>     # $ = interrupt.
> [pid 371931] <... tgkill resumed>)      = 0
> [pid 371933] futex(0xc000059148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 371931] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
> [pid 371933] <... futex resumed>)       = 1
> [pid 371932] <... futex resumed>)       = 0
> [pid 371933] write(2, "19:24:40.813294 doInterrupt\n", 28 <unfinished ...>
> 19:24:40.813294 doInterrupt
>
> A bit of browsing through the Go source code suggests that SIGURG is
> used to preempt long-running goroutines, so it could be issued more or
> less at random.
>
> Nevertheless, FUSE should also not be reissuing the reads, even if
> there were interrupts, right?

Is there a link to the test? Is it easy to repro this issue?

If I'm understanding your post correctly, the issue you are seeing is
that if your go-fuse server returns 25 entries to an interrupted
READDIRPLUS request, the kernel's next READDIRPLUS request is at
offset 1 instead of at offset 25?


Thanks,
Joanne

>
> > > operation, but go-fuse ignores the interrupt and returns 25 entries.
> > > The kernel somehow thinks that only 1 entry was consumed, and issues
> > > the next READDIRPLUS at offset 1. If go-fuse ignores the faulty offset
> > > and continues the listing (ie. continuing with entry 25), the test
> > > passes.
> > >
> > > Is this behavior of the kernel expected or a bug?
> > >
> > > I am redoing the API for directory listing to support cacheable and
> > > seekable directories, and in the new version, this looks like a
> > > directory seek. If the file system does not support seekable
> > > directories, I must return some kind of error (which is the ENOTSUP
> > > you can see in the log above).
> >
> > Is this with or without FOPEN_CACHE_DIR? Would be helpful to know
> > if FOPEN_CACHE_DIR - fuse kernel code is quite different when this
> > is set.
>
> without FOPEN_CACHE_DIR.
>
> > > I started seeing this after upgrading to Fedora 40. My kernel is
> > > 6.10.7-200.fc40.x86_64
> > >
> >
> > Would be interesting to know your kernel version before? There is
> > commit cdf6ac2a03d2, which removes a readdir lock. Although the
> > commit message explains why it is not needed anymore.
>
> The failure happens in single threaded loads, so a race condition
> seems unlikely.
>
> Let me try with other kernel versions.
>
> Also, I realize that Fedora 40 also upgraded the Go compiler, and may
> have made SIGURG be triggered more frequently?
>
> --
> Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen
>

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Wed, Sep 11, 2024 at 12:04 AM Joanne Koong <joannelkoong@gmail.com> wrote:
> > A bit of browsing through the Go source code suggests that SIGURG is
> > used to preempt long-running goroutines, so it could be issued more or
> > less at random.
> >
> > Nevertheless, FUSE should also not be reissuing the reads, even if
> > there were interrupts, right?
>
> Is there a link to the test? Is it easy to repro this issue?

I made an easy repro available over here:

  https://review.gerrithub.io/c/hanwen/go-fuse/+/1200990

To repro,

  git init
  git fetch https://review.gerrithub.io/hanwen/go-fuse
refs/changes/90/1200990/1 && git checkout FETCH_HEAD
  go test ./fs -run TestInterruptReaddirplus -count 2000

to get debug logs, add -v to the test command. Typical output:

$ go test ./fs -run TestInterruptReaddirplus -count 2000
11:42:29.186131 writer: Write/Writev failed, err: 2=no such file or
directory. opcode: RELEASEDIR
11:42:30.160136 doInterrupt
11:42:30.160559 observed seek
--- FAIL: TestInterruptReaddirplus (0.01s)
    mem_test.go:301: read back 76 entries, want 100

I am using

$ uname -a
Linux fedora 6.10.7-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 30
00:08:59 UTC 2024 x86_64 GNU/Linux
$ go version
go version go1.22.6 linux/amd64


> If I'm understanding your post correctly, the issue you are seeing is
> that if your go-fuse server returns 25 entries to an interrupted
> READDIRPLUS request, the kernel's next READDIRPLUS request is at
> offset 1 instead of at offset 25?

yes. If the offset is ignored (mustSeek = false in fs/bridge.go), it
causes test failures, because of a short read on the readdir result
there are too few entries.

If I don't ignore the offset, I have to implement a workaround on my
side which is expensive and clumsy (which is what the `mustSeek`
variable controls.)

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

> 
> 
>> If I'm understanding your post correctly, the issue you are seeing is
>> that if your go-fuse server returns 25 entries to an interrupted
>> READDIRPLUS request, the kernel's next READDIRPLUS request is at
>> offset 1 instead of at offset 25?
> 
> yes. If the offset is ignored (mustSeek = false in fs/bridge.go), it
> causes test failures, because of a short read on the readdir result
> there are too few entries.
> 
> If I don't ignore the offset, I have to implement a workaround on my
> side which is expensive and clumsy (which is what the `mustSeek`
> variable controls.)
> 

That is the part I still do not understand - what is the issue if you do
not ignore the offset? Is it maybe just the test suite that expects
offset 25?


Thanks,
Bernd

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Wed, Sep 11, 2024 at 11:51 AM Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:
> > If I don't ignore the offset, I have to implement a workaround on my
> > side which is expensive and clumsy (which is what the `mustSeek`
> > variable controls.)
> >
>
> That is the part I still do not understand - what is the issue if you do
> not ignore the offset? Is it maybe just the test suite that expects
> offset 25?

Not ignoring the offset means that I have to be prepared to support
some form of directory seeks.

Directory seeking is notoriously difficult to implement in general, so
few if any users have actually done this. If you don't have to support
directory seeks, a FS can just compile a list of entries on the
OPENDIR call, which the library can then return piecewise. This is not
correct enough to export the FS over NFS, but this works well enough
for almost any other application.

I can probably kludge up something if I remember what I sent in the
last readdirplus call, but then I would like to be really sure that I
only have to deal with the last READDIRPLUS call (or READDIR as well?
not sure.) having to be redone.

Besides being annoying to write, the kludge also takes up memory and
time on every call of readdirplus.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

On 9/11/24 12:07, Han-Wen Nienhuys wrote:
> On Wed, Sep 11, 2024 at 11:51 AM Bernd Schubert
> <bernd.schubert@fastmail.fm> wrote:
>>> If I don't ignore the offset, I have to implement a workaround on my
>>> side which is expensive and clumsy (which is what the `mustSeek`
>>> variable controls.)
>>>
>>
>> That is the part I still do not understand - what is the issue if you do
>> not ignore the offset? Is it maybe just the test suite that expects
>> offset 25?
> 
> Not ignoring the offset means that I have to be prepared to support
> some form of directory seeks.
> 
> Directory seeking is notoriously difficult to implement in general, so
> few if any users have actually done this. If you don't have to support
> directory seeks, a FS can just compile a list of entries on the
> OPENDIR call, which the library can then return piecewise. This is not
> correct enough to export the FS over NFS, but this works well enough
> for almost any other application.
> 
> I can probably kludge up something if I remember what I sent in the
> last readdirplus call, but then I would like to be really sure that I
> only have to deal with the last READDIRPLUS call (or READDIR as well?
> not sure.) having to be redone.
> 
> Besides being annoying to write, the kludge also takes up memory and
> time on every call of readdirplus.
> 

Ok, it was a bit hard to extract that information. Basically kernel
behavior doesn't match your expectations and causes overhead. As I wrote
in the evening, I think the behavior comes from static bool filldir64()
(or other filldir functions) in fs.readdir.c. Oh, I just notice I had
posted the wrong line, correct one should be here

https://elixir.bootlin.com/linux/v6.10.9/source/fs/readdir.c#L350


As you can see, that is fs/readdir.c - not fuse alone. And I guess it is
right to stop on a pending signal. For me a but surprising that the
first entry is still accepted and only then the signal is checked.
One option would be to ignore that signal in userspace before readdir
and to reset after that?


Thanks,
Bernd

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Wed, Sep 11, 2024 at 12:31 PM Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:
> Ok, it was a bit hard to extract that information. Basically kernel
> behavior doesn't match your expectations and causes overhead. As I wrote
> in the evening, I think the behavior comes from static bool filldir64()
> (or other filldir functions) in fs.readdir.c. Oh, I just notice I had
> posted the wrong line, correct one should be here
>
> https://elixir.bootlin.com/linux/v6.10.9/source/fs/readdir.c#L350

Ah, I was already wondering, as I couldn't understand why your
previous code link was relevant.

> As you can see, that is fs/readdir.c - not fuse alone. And I guess it is
> right to stop on a pending signal. For me a but surprising that the
> first entry is still accepted and only then the signal is checked.

Do you know how old this behavior is? It would be great to not have to
write the kludge on my side, but if it has been out there for a long
time, I can't pretend the problem doesn't exist once it is fixed, as
it will still crop up if folks run things on older kernels. The
runtime for Go has been issuing SIGURG for preempted goroutines since
~2020.

> One option would be to ignore that signal in userspace before readdir
> and to reset after that?

I am not sure what change you are suggesting here. Can you clarify?

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

On 9/11/24 14:08, Han-Wen Nienhuys wrote:
> On Wed, Sep 11, 2024 at 12:31 PM Bernd Schubert
> <bernd.schubert@fastmail.fm> wrote:
>> Ok, it was a bit hard to extract that information. Basically kernel
>> behavior doesn't match your expectations and causes overhead. As I wrote
>> in the evening, I think the behavior comes from static bool filldir64()
>> (or other filldir functions) in fs.readdir.c. Oh, I just notice I had
>> posted the wrong line, correct one should be here
>>
>> https://elixir.bootlin.com/linux/v6.10.9/source/fs/readdir.c#L350
> 
> Ah, I was already wondering, as I couldn't understand why your
> previous code link was relevant.
> 
>> As you can see, that is fs/readdir.c - not fuse alone. And I guess it is
>> right to stop on a pending signal. For me a but surprising that the
>> first entry is still accepted and only then the signal is checked.
> 
> Do you know how old this behavior is? It would be great to not have to
> write the kludge on my side, but if it has been out there for a long
> time, I can't pretend the problem doesn't exist once it is fixed, as
> it will still crop up if folks run things on older kernels. The
> runtime for Go has been issuing SIGURG for preempted goroutines since
> ~2020.

Following git history, I think introduced here

commit 1f60fbe7274918adb8db2f616e321890730ab7e3
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Sat Apr 23 22:50:07 2016 -0400

    ext4: allow readdir()'s of large empty directories to be interrupted


> 
>> One option would be to ignore that signal in userspace before readdir
>> and to reset after that?
> 
> I am not sure what change you are suggesting here. Can you clarify?
> 

I mean SIG_IGN, either at fuse server startup or at set in opendir and
unset in closedir, the latter is rather ugly for multi threaded fuse server.


Bernd

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Wed, Sep 11, 2024 at 3:06 PM Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:
> I mean SIG_IGN, either at fuse server startup or at set in opendir and
> unset in closedir, the latter is rather ugly for multi threaded fuse server.

This would work for my tests, where the FUSE server is the same
process as the one opening the directory, but if that is not the case,
the SIG_IGN would have to be set not on the server, but on the process
accessing the filesystem, right?

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

On 9/11/24 17:59, Han-Wen Nienhuys wrote:
> On Wed, Sep 11, 2024 at 3:06 PM Bernd Schubert
> <bernd.schubert@fastmail.fm> wrote:
>> I mean SIG_IGN, either at fuse server startup or at set in opendir and
>> unset in closedir, the latter is rather ugly for multi threaded fuse server.
> 
> This would work for my tests, where the FUSE server is the same
> process as the one opening the directory, but if that is not the case,
> the SIG_IGN would have to be set not on the server, but on the process
> accessing the filesystem, right?
> 

Oh right, won't help to do it in fuse server.

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Wed, Sep 11, 2024 at 3:06 PM Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:
> > Do you know how old this behavior is? It would be great to not have to
> > write the kludge on my side, but if it has been out there for a long
> > time, I can't pretend the problem doesn't exist once it is fixed, as
> > it will still crop up if folks run things on older kernels. The
> > runtime for Go has been issuing SIGURG for preempted goroutines since
> > ~2020.
>
> Following git history, I think introduced here
>
> commit 1f60fbe7274918adb8db2f616e321890730ab7e3
> Author: Theodore Ts'o <tytso@mit.edu>
> Date:   Sat Apr 23 22:50:07 2016 -0400
>
>     ext4: allow readdir()'s of large empty directories to be interrupted

Could the same behavior happen for readdir, or is only readdirplus affected?

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen

Re: Interrupt on readdirplus?

[Bernd Schubert]

On 9/12/24 19:26, Han-Wen Nienhuys wrote:
> On Wed, Sep 11, 2024 at 3:06 PM Bernd Schubert
> <bernd.schubert@fastmail.fm> wrote:
>>> Do you know how old this behavior is? It would be great to not have to
>>> write the kludge on my side, but if it has been out there for a long
>>> time, I can't pretend the problem doesn't exist once it is fixed, as
>>> it will still crop up if folks run things on older kernels. The
>>> runtime for Go has been issuing SIGURG for preempted goroutines since
>>> ~2020.
>>
>> Following git history, I think introduced here
>>
>> commit 1f60fbe7274918adb8db2f616e321890730ab7e3
>> Author: Theodore Ts'o <tytso@mit.edu>
>> Date:   Sat Apr 23 22:50:07 2016 -0400
>>
>>      ext4: allow readdir()'s of large empty directories to be interrupted
> 
> Could the same behavior happen for readdir, or is only readdirplus affected?
> 

Should be the same, it calls fuse_emit(), which calls into the filldir 
functions in fs/readdir.c.

A way out might be a cached readdir (FOPEN_CACHE_DIR) as the entries do 
not get lost.


Bernd

Re: Interrupt on readdirplus?

[Han-Wen Nienhuys]

On Wed, Sep 11, 2024 at 3:06 PM Bernd Schubert
<bernd.schubert@fastmail.fm> wrote:

> >> As you can see, that is fs/readdir.c - not fuse alone. And I guess it is
> >> right to stop on a pending signal. For me a but surprising that the
> >> first entry is still accepted and only then the signal is checked.
> >
> > Do you know how old this behavior is? It would be great to not have to
> > write the kludge on my side, but if it has been out there for a long
> > time, I can't pretend the problem doesn't exist once it is fixed, as
> > it will still crop up if folks run things on older kernels. The
> > runtime for Go has been issuing SIGURG for preempted goroutines since
> > ~2020.
>
> Following git history, I think introduced here
>
> commit 1f60fbe7274918adb8db2f616e321890730ab7e3
> Author: Theodore Ts'o <tytso@mit.edu>
> Date:   Sat Apr 23 22:50:07 2016 -0400

yeah, that is too old. I implemented the kludge here:

https://review.gerrithub.io/c/hanwen/go-fuse/+/1201139

It is not that bad after all.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen