[Bug 201555] New: [f2fs] Processes freezing periodically with f2fs root

[Bug 201555] New: [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

            Bug ID: 201555
           Summary: [f2fs] Processes freezing periodically with f2fs root
           Product: File System
           Version: 2.5
    Kernel Version: 4.14 - 4.19 (at least)
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: f2fs
          Assignee: filesystem_f2fs@kernel-bugs.kernel.org
          Reporter: gondsman@techgeek.co.in
        Regression: No

I'm running Manjaro linux on two separate machines with very different hardware
and both experience processes (mostly systemd-journal) freeze after long-ish
uptimes (sometimes hours, sometimes days). After that the system eventually
becomes unresponsive and a hard reset is needed.

I assume this is related to the fact that I have f2fs as a root partition on
both machines. I can read something like this in my dmesg, repeating every 120
seconds:

[322559.191681] INFO: task systemd-journal:260 blocked for more than 120
seconds.
[322559.191692]       Tainted: P           O    4.14.74-1-MANJARO #1
[322559.191695] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[322559.191700] systemd-journal D    0   260      1 0x00000104
[322559.191706] Call Trace:
[322559.191720]  ? __schedule+0x286/0x890
[322559.191725]  schedule+0x2f/0x90
[322559.191731]  rwsem_down_read_failed+0xe3/0x150
[322559.191738]  ? dput.part.35+0x28/0x1e0
[322559.191744]  call_rwsem_down_read_failed+0x14/0x30
[322559.191750]  down_read+0x13/0x30
[322559.191770]  f2fs_vm_page_mkwrite+0x72/0x550 [f2fs]
[322559.191778]  do_page_mkwrite+0x31/0x90
[322559.191783]  do_wp_page+0x400/0x560
[322559.191789]  ? _copy_to_user+0x26/0x30
[322559.191796]  ? cp_new_stat+0x150/0x180
[322559.191801]  __handle_mm_fault+0xbf2/0x1520
[322559.191809]  handle_mm_fault+0xaa/0x1d0
[322559.191817]  __do_page_fault+0x258/0x530
[322559.191825]  ? page_fault+0x2f/0x50
[322559.191831]  page_fault+0x45/0x50
[322559.191836] RIP: b737dfa0:0x559314bf7c60
[322559.191839] RSP: 0050:0000000000000001 EFLAGS: 7ffeb737dfa8
[322601.532195] systemd[1]: systemd-journald.service: State 'stop-sigabrt'
timed out. Terminating.

I can rule out a hardware issue because I even swapped disks on one of my
machines and it didn't solve it. It happens on both SATA and NVMe drives. I
tried every kernel version from 4.14 to 4.19 and it didn't make a difference.
I've read bug 195983 as my issue looked similar to it, but adding nomerge_flush
mount flag also had no effect.

Unfortunately it's pretty difficult to reproduce the bug consistently, as it
can take literally days for it to show up.

Can I do anything else to help debugging it?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201555] [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

Chao Yu (chao@kernel.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |chao@kernel.org

--- Comment #1 from Chao Yu (chao@kernel.org) ---
Hi,

Could you please have a try with last f2fs in below dev branch?

https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/log/?h=dev

And could you attach entire dmesg file?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201555] [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

--- Comment #2 from Pierpaolo Valerio (gondsman@techgeek.co.in) ---
I'll try the dev branch as soon as I get the time to do it. I can attach the
full dmesg I got a few days ago for the time being.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201555] [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

--- Comment #3 from Pierpaolo Valerio (gondsman@techgeek.co.in) ---
Created attachment 279271
  --> https://bugzilla.kernel.org/attachment.cgi?id=279271&action=edit
dmesg on 4.14 kernel (but it's the same on newer versions)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201555] [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

--- Comment #4 from Pierpaolo Valerio (gondsman@techgeek.co.in) ---
Ok, I've compiled your dev branch and I'll report back in a few days (or
earlier if the PC freezes again). Thanks!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201555] [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

--- Comment #5 from Pierpaolo Valerio (gondsman@techgeek.co.in) ---
So, my office PC is still up after more than 5 days, so I'm assuming the
problem is either solved or greatly reduced.
When can I expect to see the changes merged? Will it be in a point release or
should I wait for 4.20?
Thanks a lot.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 201555] [f2fs] Processes freezing periodically with f2fs root

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=201555

Chao Yu (chao@kernel.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #6 from Chao Yu (chao@kernel.org) ---
(In reply to Pierpaolo Valerio from comment #5)
> So, my office PC is still up after more than 5 days, so I'm assuming the
> problem is either solved or greatly reduced.

I guess below two commits may fix potential deadlock issue:

1) Revert: "f2fs: check last page index in cached bio to decide submission"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bab475c5414e8d1fa182fd17ae966864e9c85741

2) f2fs: submit cached bio to avoid endless PageWriteback
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=48018b4cfd07dd2df9a067fb3a6a3221c19eed11

I think you can just merge them to stable kernel with the specified version as
you wish, and then have a try.

> When can I expect to see the changes merged? Will it be in a point release

It has been merged into Linux 4.20-rc1, maybe that can be your option if you
don't want to wait 4.20 about 2 months bug fixing iteration.

> or should I wait for 4.20?
> Thanks a lot.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.