ublk: Graceful Upgrade of ublk server application

ublk: Graceful Upgrade of ublk server application

[Yoav Cohen]

I am seeking advice on whether it is possible to upgrade the ublksrv version without terminating the daemon abruptly. Specifically, I would like the daemon to exit gracefully, ensuring all necessary cleanups are performed.
In my current implementation, I attempted to cancel all ublk uring SQEs (specifically the COMMIT_AND_FETCH_REQ/FETCH_REQ operations) using the following approach:
io_uring_prep_cancel_fd(sqe, cdev_fd, IORING_ASYNC_CANCEL_ALL);io_uring_prep_cancel_fd(sqe, cdev_fd, IORING_ASYNC_CANCEL_ALL);
However, this method does not seem to be effective. In my scenario, I have a single io_uring instance that serves multiple devices and other producers, so simply stopping the polling of CQEs is not a viable solution due to potential race conditions.
Could you please provide any suggestions or guidance on how to achieve a graceful upgrade of the ublksrv version without causing disruptions?
 
Thank you

Re: ublk: Graceful Upgrade of ublk server application

[Ming Lei]

On Tue, Apr 15, 2025 at 08:15:08AM +0000, Yoav Cohen wrote:
> I am seeking advice on whether it is possible to upgrade the ublksrv version without terminating the daemon abruptly. Specifically, I would like the daemon to exit gracefully, ensuring all necessary cleanups are performed.
> In my current implementation, I attempted to cancel all ublk uring SQEs (specifically the COMMIT_AND_FETCH_REQ/FETCH_REQ operations) using the following approach:
> io_uring_prep_cancel_fd(sqe, cdev_fd, IORING_ASYNC_CANCEL_ALL);io_uring_prep_cancel_fd(sqe, cdev_fd, IORING_ASYNC_CANCEL_ALL);

That shouldn't work because COMMIT_AND_FETCH_REQ/FETCH_REQ has been
issued to ublk driver already.

> However, this method does not seem to be effective. In my scenario, I have a single io_uring instance that serves multiple devices and other producers, so simply stopping the polling of CQEs is not a viable solution due to potential race conditions.
> Could you please provide any suggestions or guidance on how to achieve a graceful upgrade of the ublksrv version without causing disruptions?
>  

You can delete all ublk devices handled by this single io_ring instance
first by sending UBLK_CMD_DEL_DEV, then exit ublk server loop if there
isn't any pending uring_cmd & target IO. And the ublk server need to stop
to issue COMMIT_AND_FETCH_REQ after getting uring_cmd with UBLK_IO_RES_ABORT.

I guess you want to send the control command in single pthread too?

If yes, it still can work with coroutine or modern language's .async/await.

This feature is actually in my todo list for libublk-rs, just not started
because of not seeing real requirement.


Thanks,
Ming

Re: ublk: Graceful Upgrade of ublk server application

[Yoav Cohen]

Hi Ming,

Thank you for the fast reply.
To be clear, I don't want calling DELETE_DEV or STOP_DEV as I want the kernel bdev will be stay while upgrading the ublk server application.
It would be nice to have a nice way to have something like FREEZE_DEV that we may use which will also make all the cmds back with ABORT result but both block and char device will be stay until a new userspace application will reconnect.

Thank you

________________________________________
From: Ming Lei <ming.lei@redhat.com>
Sent: Tuesday, April 15, 2025 2:06 PM
To: Yoav Cohen
Cc: linux-block@vger.kernel.org; axboe@kernel.dk
Subject: Re: ublk: Graceful Upgrade of ublk server application

External email: Use caution opening links or attachments


On Tue, Apr 15, 2025 at 08:15:08AM +0000, Yoav Cohen wrote:
> I am seeking advice on whether it is possible to upgrade the ublksrv version without terminating the daemon abruptly. Specifically, I would like the daemon to exit gracefully, ensuring all necessary cleanups are performed.
> In my current implementation, I attempted to cancel all ublk uring SQEs (specifically the COMMIT_AND_FETCH_REQ/FETCH_REQ operations) using the following approach:
> io_uring_prep_cancel_fd(sqe, cdev_fd, IORING_ASYNC_CANCEL_ALL);io_uring_prep_cancel_fd(sqe, cdev_fd, IORING_ASYNC_CANCEL_ALL);

That shouldn't work because COMMIT_AND_FETCH_REQ/FETCH_REQ has been
issued to ublk driver already.

> However, this method does not seem to be effective. In my scenario, I have a single io_uring instance that serves multiple devices and other producers, so simply stopping the polling of CQEs is not a viable solution due to potential race conditions.
> Could you please provide any suggestions or guidance on how to achieve a graceful upgrade of the ublksrv version without causing disruptions?
>

You can delete all ublk devices handled by this single io_ring instance
first by sending UBLK_CMD_DEL_DEV, then exit ublk server loop if there
isn't any pending uring_cmd & target IO. And the ublk server need to stop
to issue COMMIT_AND_FETCH_REQ after getting uring_cmd with UBLK_IO_RES_ABORT.

I guess you want to send the control command in single pthread too?

If yes, it still can work with coroutine or modern language's .async/await.

This feature is actually in my todo list for libublk-rs, just not started
because of not seeing real requirement.


Thanks,
Ming

Re: ublk: Graceful Upgrade of ublk server application

[Uday Shankar]

On Tue, Apr 15, 2025 at 07:46:51PM +0000, Yoav Cohen wrote:
> Hi Ming,
> 
> Thank you for the fast reply.
> To be clear, I don't want calling DELETE_DEV or STOP_DEV as I want the kernel bdev will be stay while upgrading the ublk server application.
> It would be nice to have a nice way to have something like FREEZE_DEV that we may use which will also make all the cmds back with ABORT result but both block and char device will be stay until a new userspace application will reconnect.

Have you taken a look at the recovery flags? These offer slightly
different behaviors around how I/O is handled while the ublk server is
dying/when it is dead, but they all keep the block device up even after
the ublk server exits.

The flags are documented at https://docs.kernel.org/block/ublk.html

Re: ublk: Graceful Upgrade of ublk server application

[Ming Lei]

On Tue, Apr 15, 2025 at 06:51:57PM -0600, Uday Shankar wrote:
> On Tue, Apr 15, 2025 at 07:46:51PM +0000, Yoav Cohen wrote:
> > Hi Ming,
> > 
> > Thank you for the fast reply.

oops, looks I didn't get your reply, :-(

> > To be clear, I don't want calling DELETE_DEV or STOP_DEV as I want the kernel bdev will be stay while upgrading the ublk server application.

Can you explain a bit what is upgrading? Is it simple application binary
replacement?

> > It would be nice to have a nice way to have something like FREEZE_DEV that we may use which will also make all the cmds back with ABORT result but both block and char device will be stay until a new userspace application will reconnect.
> 

Looks one reasonable requirement, maybe SUSPEND_DEV and RESUME_DEV command,
which exists on RAID/DM too.

Also when device is in (new)suspended state, parameter can be re-configured.

Most of recover code can be reused, in theory it shouldn't be hard to
support, but one trouble could be what if both uring exiting and SUSPEND_DEV
happen at the same time? This corner case need to be take into account. 

I'd suggest to cover more requirements given it should be one generic
interface.

> Have you taken a look at the recovery flags? These offer slightly
> different behaviors around how I/O is handled while the ublk server is
> dying/when it is dead, but they all keep the block device up even after
> the ublk server exits.
> 
> The flags are documented at https://docs.kernel.org/block/ublk.html

The recovery mechanism is triggered passively, and here the upgrading
needs to suspend device voluntarily & gracefully.

Maybe add one control command of COOP_CANCEL_FOR_RECOVERY or ABORT_URING_CMD
to abort active uring_cmd for triggering recovery from userspace? Which looks
much easier. But it need cooperation between ublk driver and server.


Thanks,
Ming

Re: ublk: Graceful Upgrade of ublk server application

[Yoav Cohen]

Hi,

The use case is as you say to replace the binary (update) without making the bdev to disappear.
Currently I don't even use the user_copy(to avoid the 1 more system call) so the io buffer is also part of the sqe which is prevent me from free it from userspace perspective.
So yes, even ABORT_URING_CMD by given tag can be enough.
What do you think?


Thank you
________________________________________
From: Ming Lei <ming.lei@redhat.com>
Sent: Wednesday, April 16, 2025 4:39 AM
To: Uday Shankar
Cc: Yoav Cohen; linux-block@vger.kernel.org; axboe@kernel.dk
Subject: Re: ublk: Graceful Upgrade of ublk server application

External email: Use caution opening links or attachments


On Tue, Apr 15, 2025 at 06:51:57PM -0600, Uday Shankar wrote:
> On Tue, Apr 15, 2025 at 07:46:51PM +0000, Yoav Cohen wrote:
> > Hi Ming,
> >
> > Thank you for the fast reply.

oops, looks I didn't get your reply, :-(

> > To be clear, I don't want calling DELETE_DEV or STOP_DEV as I want the kernel bdev will be stay while upgrading the ublk server application.

Can you explain a bit what is upgrading? Is it simple application binary
replacement?

> > It would be nice to have a nice way to have something like FREEZE_DEV that we may use which will also make all the cmds back with ABORT result but both block and char device will be stay until a new userspace application will reconnect.
>

Looks one reasonable requirement, maybe SUSPEND_DEV and RESUME_DEV command,
which exists on RAID/DM too.

Also when device is in (new)suspended state, parameter can be re-configured.

Most of recover code can be reused, in theory it shouldn't be hard to
support, but one trouble could be what if both uring exiting and SUSPEND_DEV
happen at the same time? This corner case need to be take into account.

I'd suggest to cover more requirements given it should be one generic
interface.

> Have you taken a look at the recovery flags? These offer slightly
> different behaviors around how I/O is handled while the ublk server is
> dying/when it is dead, but they all keep the block device up even after
> the ublk server exits.
>
> The flags are documented at https://docs.kernel.org/block/ublk.html

The recovery mechanism is triggered passively, and here the upgrading
needs to suspend device voluntarily & gracefully.

Maybe add one control command of COOP_CANCEL_FOR_RECOVERY or ABORT_URING_CMD
to abort active uring_cmd for triggering recovery from userspace? Which looks
much easier. But it need cooperation between ublk driver and server.


Thanks,
Ming

Re: ublk: Graceful Upgrade of ublk server application

[Ming Lei]

On Wed, Apr 16, 2025 at 08:16:44AM +0000, Yoav Cohen wrote:
> Hi,
> 
> The use case is as you say to replace the binary (update) without making the bdev to disappear.
> Currently I don't even use the user_copy(to avoid the 1 more system call) so the io buffer is also part of the sqe which is prevent me from free it from userspace perspective.
> So yes, even ABORT_URING_CMD by given tag can be enough.
> What do you think?

I think the requirement is reasonable, which could be one QUIESCE_DEV command:

- only usable for UBLK_F_USER_RECOVERY

- need ublk server cooperation for handling inflight IO command

- fallback to normal cancel code path in case that io_uring is exiting

The implementation shouldn't be hard:

- mark ubq->canceling as ture 
	- freeze request queue
	- mark ubq->canceling as true
	- unfreeze request queue

- canceling all uring_cmd with UBLK_IO_RES_ABORT (*)
	- now there can't be new ublk IO request coming, and ublk server won't
	send new uring_cmd too, 

	- the gatekeeper code of __ublk_ch_uring_cmd() should be reliable to prevent
	any new uring_cmd from malicious application, maybe need audit & refactoring
	a bit

	- need ublk server to handle UBLK_IO_RES_ABORT correctly: release all
	  kinds resource, close ublk char device...

- wait until ublk char device is released by checking UB_STATE_OPEN

- now ublk state becomes UBLK_S_DEV_QUIESCED or UBLK_S_DEV_FAIL_IO,
and userspace can replace the binary and recover device with new
application via UBLK_CMD_START_USER_RECOVERY & UBLK_CMD_END_USER_RECOVERY

Please let us know if the above works for your requirement.

Thanks,
Ming

Re: ublk: Graceful Upgrade of ublk server application

[Yoav Cohen]

Hi Ming,

Thank you very much!
The above seems to match our requirements.
Just to be sure, Do you want me to Implement it and issue a patch or do you plan add it to your plan?

Thanks

________________________________________
From: Ming Lei <ming.lei@redhat.com>
Sent: Wednesday, April 16, 2025 12:12 PM
To: Yoav Cohen
Cc: Uday Shankar; linux-block@vger.kernel.org; axboe@kernel.dk
Subject: Re: ublk: Graceful Upgrade of ublk server application

External email: Use caution opening links or attachments


On Wed, Apr 16, 2025 at 08:16:44AM +0000, Yoav Cohen wrote:
> Hi,
>
> The use case is as you say to replace the binary (update) without making the bdev to disappear.
> Currently I don't even use the user_copy(to avoid the 1 more system call) so the io buffer is also part of the sqe which is prevent me from free it from userspace perspective.
> So yes, even ABORT_URING_CMD by given tag can be enough.
> What do you think?

I think the requirement is reasonable, which could be one QUIESCE_DEV command:

- only usable for UBLK_F_USER_RECOVERY

- need ublk server cooperation for handling inflight IO command

- fallback to normal cancel code path in case that io_uring is exiting

The implementation shouldn't be hard:

- mark ubq->canceling as ture
        - freeze request queue
        - mark ubq->canceling as true
        - unfreeze request queue

- canceling all uring_cmd with UBLK_IO_RES_ABORT (*)
        - now there can't be new ublk IO request coming, and ublk server won't
        send new uring_cmd too,

        - the gatekeeper code of __ublk_ch_uring_cmd() should be reliable to prevent
        any new uring_cmd from malicious application, maybe need audit & refactoring
        a bit

        - need ublk server to handle UBLK_IO_RES_ABORT correctly: release all
          kinds resource, close ublk char device...

- wait until ublk char device is released by checking UB_STATE_OPEN

- now ublk state becomes UBLK_S_DEV_QUIESCED or UBLK_S_DEV_FAIL_IO,
and userspace can replace the binary and recover device with new
application via UBLK_CMD_START_USER_RECOVERY & UBLK_CMD_END_USER_RECOVERY

Please let us know if the above works for your requirement.

Thanks,
Ming

Re: ublk: Graceful Upgrade of ublk server application

[Ming Lei]

Hi Yoav,

On Sun, Apr 20, 2025 at 08:57:23AM +0000, Yoav Cohen wrote:
> Hi Ming,
> 
> Thank you very much!
> The above seems to match our requirements.
> Just to be sure, Do you want me to Implement it and issue a patch or do you plan add it to your plan?

It is great if you'd like to implement the feature, and please add one
selftest case(add quiesce command & one test case) together with the
driver change.

Otherwise, I can add it to my todo list.

Thanks,
Ming

> 
> Thanks
> 
> ________________________________________
> From: Ming Lei <ming.lei@redhat.com>
> Sent: Wednesday, April 16, 2025 12:12 PM
> To: Yoav Cohen
> Cc: Uday Shankar; linux-block@vger.kernel.org; axboe@kernel.dk
> Subject: Re: ublk: Graceful Upgrade of ublk server application
> 
> External email: Use caution opening links or attachments
> 
> 
> On Wed, Apr 16, 2025 at 08:16:44AM +0000, Yoav Cohen wrote:
> > Hi,
> >
> > The use case is as you say to replace the binary (update) without making the bdev to disappear.
> > Currently I don't even use the user_copy(to avoid the 1 more system call) so the io buffer is also part of the sqe which is prevent me from free it from userspace perspective.
> > So yes, even ABORT_URING_CMD by given tag can be enough.
> > What do you think?
> 
> I think the requirement is reasonable, which could be one QUIESCE_DEV command:
> 
> - only usable for UBLK_F_USER_RECOVERY
> 
> - need ublk server cooperation for handling inflight IO command
> 
> - fallback to normal cancel code path in case that io_uring is exiting
> 
> The implementation shouldn't be hard:
> 
> - mark ubq->canceling as ture
>         - freeze request queue
>         - mark ubq->canceling as true
>         - unfreeze request queue
> 
> - canceling all uring_cmd with UBLK_IO_RES_ABORT (*)
>         - now there can't be new ublk IO request coming, and ublk server won't
>         send new uring_cmd too,
> 
>         - the gatekeeper code of __ublk_ch_uring_cmd() should be reliable to prevent
>         any new uring_cmd from malicious application, maybe need audit & refactoring
>         a bit
> 
>         - need ublk server to handle UBLK_IO_RES_ABORT correctly: release all
>           kinds resource, close ublk char device...
> 
> - wait until ublk char device is released by checking UB_STATE_OPEN
> 
> - now ublk state becomes UBLK_S_DEV_QUIESCED or UBLK_S_DEV_FAIL_IO,
> and userspace can replace the binary and recover device with new
> application via UBLK_CMD_START_USER_RECOVERY & UBLK_CMD_END_USER_RECOVERY
> 
> Please let us know if the above works for your requirement.
> 
> Thanks,
> Ming
> 

-- 
Ming