Re: [bug report] rbd unmap hangs after pausing and unpausing I/O

[Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>]

Hi Raphael,

On Tue, 2025-09-23 at 12:38 +0200, Raphael Zimmer wrote:
> Hello,
> 
> I encountered an error with the kernel Ceph client (specifically using 
> an RBD device) when pausing I/O on the cluster by setting and unsetting 
> pauserd and pausewr flags. An error was seen with two different setups, 
> which I believe is due to the same problem.
> 

Thanks a lot for the report. Could you please create the ticket in a tracker
system [1]?

> 1) When pausing and later unpausing I/O on the cluster, everything seems 
> to work as expected until trying to unmap an RBD device from the kernel. 
> In this case, the rbd unmap command hangs and also can't be killed. To 
> get back to a normally working state, a system reboot is needed. This 
> behavior was observed on different systems (Debian 12 and 13) and could 
> also be reproduced with an installation of the mainline kernel (v6.17-rc6).
> 
> Steps to reproduce:
> - Connect kernel client to RBD device (rbd map)
> - Pause I/O on cluster (ceph osd pause)
> - Wait some time (3 minutes should be enough)
> - Unpause I/O on cluster
> - Try to unmap RBD device on client
> 

Do you have a script? Could you please share the sequence of commands that you
used in command line to reproduce the issue?

Have you created any folders/files before pause/unpause the I/O requests on
cluster?
How have you initiated the I/O operations before pausing the I/O requests on
cluster?
Have you observed any warnings, call traces, or crashes from CephFS kernel
client in system log when rbd unmap command hangs (usually, kernel complains if
something is hanging significant amount of time)?

Thanks,
Slava.

> 
> 2) When using an application that internally uses the kernel Ceph client 
> code, I observed the following behavior:
> 
> Pausing I/O leads to a watch error after some time (same as with failing 
> OSDs or e.g. when pool quota is reached). In rbd_watch_errcb 
> (drivers/block/rbd.c), the watch_dwork gets scheduled, which leads to a 
> call of rbd_reregister_watch -> __rbd_register_watch -> ceph_osdc_watch 
> (net/ceph/osd_client.c) -> linger_reg_commit_wait -> 
> wait_for_completion_killable. At this point, it waits without any 
> timeout for the completion. The normal behavior is to wait until the 
> causing condition is resolved and then return. With pausing and 
> unpausing I/O, wait_for_completion_killable does not return even after 
> unpausing because no call to complete or complete_all happens. I would 
> guess that on unpausing some call is missing so that committing the 
> linger request never completes.
> 
>  From what I am seeing, it seems like this missing completion in the 
> second case is also the cause of the hanging rbd unmap with the 
> unmodified kernel.
> 
> 
> Best regards,
> 
> Raphael

[1] https://tracker.ceph.com