Re: Optimizing C/R Image Format for Kubernetes

[Radostin Stoyanov <rstoyanov1@gmail.com>]

Hi Andrei,

Thank you for investing! I like the idea about "Directly Accessible CRIU 
Images". This was also one of the reasons we chose to implement 
encryption support directly within the checkpoint/restore operations in 
CRIU, rather than encrypting the tar archives. Enabling support for 
encryption is an important requirement for the checkpoint/restore 
functionality in Kubernetes. I will rebase my patches on the criu-dev 
branch and open a pull request.

Best wishes,
Radostin


On 19/06/2025 00:58, Andrei Vagin wrote:
> Hi everyone,
>
> I've been spending the last few days diving into checkpoint/restore
> (C/R) within Kubernetes, specifically focusing on the restore process
> and the current image format.
>
> I found the current container image format to be suboptimal. I've
> examined containerd, and I suspect CRI-O has similar issues.
> Essentially, it's a container image that encapsulates a
> checkpoint-restore archive. Each container start requires multiple
> unpacking steps:
> * Extracting the C/R archive: This yields two tar archives—one for the
>    filesystem delta and another for CRIU images.
> * Applying the filesystem delta: We need to mount the container's root
>    filesystem, then extract and apply this delta.
> * Restoring the container: Finally, we extract the CRIU images and
>    proceed with the restore.
>
> I believe this format, with its nested tar archives, leads to a
> significant amount of time wasted on unpacking, which directly impacts
> performance.
>
> With the growing interest in using C/R to optimize application startup
> time. I've run some experiments. My findings indicate that the current
> image format significantly reduces the benefits of C/R, and in many
> cases, restoring a container from these images is actually slower than
> starting it from scratch.
>
> Here's my vision for an ideal image format for C/R-ed containers:
> * Filesystem Delta as an Overlay Layer: The filesystem delta should be
>    treated just like any other container image delta. This means it would
>    be specified as one of the overlay layers when a container is mounted.
> * Directly Accessible CRIU Images: Once an image is pulled locally, the
>    CRIU images should not be bundled in a tar archive. Instead, they
>    should be placed directly in a directory, allowing CRIU to use them
>    immediately without any extra extraction steps.
>
> Thanks,
> Andrei