Re: Re: RE: Re: [PATCH] ceph: fix potential stray locked folios during umount

李磊
2026-04-29T14:42:50Z

> 2026年4月28日 05:52，Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> 写道：
>
> 安全提示：此邮件来自公司外部。除非您确认发件人身份可信且邮件内容不含可疑信息，否则请勿回复或转发邮件、点击邮件链接或打开附件。
>
>
> On Sun, 2026-04-26 at 15:38 +0000, 李磊 wrote:
>>
>>>>
>
> <skipped>
>
>>>> I understand your concern. This patch is a truly straightforward workaround.
>>>> So, how about we just abort OSD requests if they take too long to return
>>>> during unmounting ?
>>>
>>> The question here is how to define that OSD requests taking too long time?
>>> Potentially, processing could be really slow for some reason. From one point of
>>> view, if we know that destination OSD is down or we have network partitioning,
>>> then it doesn't make sense to wait to long. I am thinking about potential
>>> checking of number of OSD requests. If this number is going down, then it needs
>>> to wait, otherwise, if this number doesn't change, then it needs to finish the
>>> unmount without waiting. Does it make sense?
>>>
>>>>
>>>> Compared to leaving some locked folios in the system, return -EIO to those
>>>> OSD requests which may never return is more reasonable. This is because locked
>>>> folios left behind Cephfs unmount may block kcompactd and render the entire
>>>> system unstable.
>>>>
>>>
>>> I agree. It makes sense. If we know that some OSD requests will never return,
>>> then we need to manage this situation in better way. But how could we detect
>>> that OSD request will never return?
>>>
>>>> Besides, successful unmounting doesn't guarantee dirty buffers are successfully
>>>> written to the backend. For example, when a buffered write returns, the local
>>>> filesystem may encounter bad blocks on the local disk and -EIO is returned to
>>>> the writeback kworkers. Therefore, in our scenario, does it make sense if we
>>>> treat the OSD requests that have been flight for a certain period as failed,
>>>> And return -EIO to the caller?
>>>
>>> This is the main question: how to detect that OSD requests are failed?
>>>
>>> As far as I can see, if an OSD is down and osd_request_timeout is not set (the
>>> default), a stalled write can block unmount indefinitely. I assume that you have
>>> the osd_request_timeout is not set. So, maybe, we need to re-consider the policy
>>> of management the stuck OSD requests during unmount.
>>>
>>> Laggy OSD path: if any request's r_stamp is older than osd_keepalive_timeout,
>>> the OSD goes on a slow_osds list and ceph_con_keepalive() is called, sending a
>>> keepalive byte over TCP. If the TCP connection is silently broken, the keepalive
>>> write will fail, triggering con_fault().
>>>
>>> Timed-out request path: if osd_request_timeout is set (default 0 = disabled),
>>> requests older than that deadline are aborted with -ETIMEDOUT via
>>> abort_request().
>>>
>>> Homeless requests: requests that can't be mapped to any OSD are also checked
>>> against osd_request_timeout.
>>>
>>> The ceph_con_keepalive_expired() uses the timestamp of the last keepalive
>>> acknowledgement (con->last_keepalive_ack) to determine whether the peer has gone
>>> silent beyond interval. When this fires, the connection is considered dead and
>>> con_fault() is triggered.
>>>
>>> So, we need to find a proper approach of finding a good solution from available
>>> functionality.
>>
>> I agree. Instead of waiting for inflight requests infinitely or aborting OSD
>> requests brutally, you prefer a much more elegant way to deal with this dilemma.
>> It’s cool, but it seems complex and more time is needed to fix locked folios leakage
>> on the client nodes. Is there any acceptable short-term scheme?
>
> Have you tried to set up the osd_request_timeout and to see how CephFS kernel
> client will behave afterwards? Will it change anything?

If I apply this patch to wait for stopping blockers to drop to zero, setting osd_request_timeout
can help abort OSD requests in time and allow the unmount process to proceed. However
I think we still have 2 aspects to discuss.

1. Instead of using mount_timeout, can we use other option to accommodate waiting during
the unmount process?

It is somewhat confusing that the mount_timeout option decides how long we should wait
for both dirty_folios and stopping_blockers if they don’t drop to zero. As for as I know
mount_timeout determines the maximum wait time in open_root_dentry() for loading root
inode during the mount operation.

Just for the scenario I described — stop all the OSDs and kill buffered read, is it
better to use osd_request_timeout instead?

Or can we wait_for_completion() infinitely if an OSD request never returns, but create a
debugfs file (for example ‘abort’) to tigger all OSD’s requests to ensure a clean and
successful and unmount.

2. Is killable waiting really suitable here ?

Any user-space process may send a kill signal to the unmount process, which may leave
behind some stray locked folios and degrade the system stability. Maybe we should use
non-killable functions here ?

Thanks,
Li

>
>>
>> I find it is not easy to work around this issue by merely increasing opt->mount_timeout.
>> Both dirty_folios and stopping_blockers wait with TASK_KILLABLE set, which means the
>> unmount process’s wait can be interrupted by a kill signal and leave some locked folios
>> after unmount regardless of the mount_timeout setting.
>
> If somebody (or something) would like to kill the process, then there is nothing
> that we can do. The potential kill signal can be received at any time point and
> some locked folios continue to exists.
>
>>
>>>
>>>>
>>>> Lastly, I think we can just use stopping blockers to replace dirty_folios to
>>>> simplify the unmounting and wait process. Accordingly, in ceph_kill_sb(), we
>>>> only need to wait for stopping_blockers count to drop to zero. If a timeout
>>>> occurs, we can cancel all the inflight requests and print some warning messages.
>>>
>>> The dirty_folios is "how much dirty data is still to be flushed," while
>>> stopping_blockers is "how many threads are currently inside code that holds an
>>> implicit reference to the MDS client." Unmount must drain both in order, and the
>>> two counters solve entirely different races.
>>>
>>> The mdsc->dirty_folios — count of dirty page-cache folios not yet written back.
>>> It incremented in ceph_dirty_folio() at the moment of folio transitions from
>>> clean to dirty in the page cache. It decremented in the OSD write-completion
>>> callback after the OSD acknowledges the writeback and end_page_writeback() is
>>> called. It represents the number of file-data folios that have been dirtied
>>> (modified in the page cache) but whose data has not yet reached an OSD (i.e.,
>>> writeback is pending or in flight).
>>>
>>> The mdsc->stopping_blockers counts of in-progress MDS/OSD message handlers. It
>>> incremented by ceph_inc_mds_stopping_blocker() / ceph_inc_osd_stopping_blocker()
>>> at the entry of any async operation that must not be interrupted mid-flight by
>>> shutdown. It decremented at the exit of the async operations' handlers.
>>
>> Thanks for the detailed explanation of dirty_folios and stopping_blockers. However,
>> it is still a bit confusing that we need to wait dirty foios to decrease to zero
>> after sync_filesystem() in ceph_kill_sb(). It seems that the semantics of
>> sync_filesystem() is broken in Cephfs? Because in the common situation, sync_filesystem()
>> guarantees dirty folios are flushed to the backend, and PG_dirty of these folios
>> are cleared after it returns.
>>
>>
>
> As far as I can see, during sync_filesystem() call, it is used
> filemap_fdatawait_keep_errors() that waits for pages tagged
> PAGECACHE_TAG_WRITEBACK. A dirty page that was found by ceph_writepages_start()
> but never submitted stays tagged PAGECACHE_TAG_DIRTY, and sync_filesystem()
> returns treating it as invisible. So, this is why we need to wait dirty foios to
> decrease to zero in ceph_kill_sb().
>
> Thanks,
> Slava.