Re: Optimizing C/R Image Format for Kubernetes

[Adrian Reber <areber@redhat.com>]

On Fri, Jun 20, 2025 at 12:34:22PM -0700, Andrei Vagin wrote:
> On Thu, Jun 19, 2025 at 4:06 AM Adrian Reber <areber@redhat.com> wrote:
> ...
> >
> > > Here's my vision for an ideal image format for C/R-ed containers:
> > > * Filesystem Delta as an Overlay Layer: The filesystem delta should be
> > >   treated just like any other container image delta. This means it would
> > >   be specified as one of the overlay layers when a container is mounted.
> >
> > Yes. The current format was my wrong decision as I was not familiar
> > with how those delta layers are working.
> >
> > > * Directly Accessible CRIU Images: Once an image is pulled locally, the
> > >   CRIU images should not be bundled in a tar archive. Instead, they
> > >   should be placed directly in a directory, allowing CRIU to use them
> > >   immediately without any extra extraction steps.
> >
> > This is not actually true. The OCI image does not contain the tar
> > archive but the actual checkpoint files directly:
> >
> > # podman pull quay.io/adrianreber/checkpoint-test:tag73
> > Trying to pull quay.io/adrianreber/checkpoint-test:tag73...
> > Getting image source signatures
> > Copying blob e65839d7ec1b done
> > Copying config 27d63848a3 done
> > Writing manifest to image destination
> > Storing signatures
> > 27d63848a32d24c68b131f99880411c11af6519820ef22b989a86b7f10038c79
> > # podman image mount quay.io/adrianreber/checkpoint-test:tag73
> > /var/lib/containers/storage/overlay/98aaf3c7dc28cfb2e79893ef952380b00169dcce910be48bbea1143b07ae2a0e/merged
> > # ls -la /var/lib/containers/storage/overlay/98aaf3c7dc28cfb2e79893ef952380b00169dcce910be48bbea1143b07ae2a0e/merged
> > total 44
> > dr-xr-xr-x. 1 root root  4096 Jun 19 10:53 .
> > drwx------. 6 root root  4096 Jun 19 10:53 ..
> > -rw-------. 1 root root  1120 Feb  1 11:11 bind.mounts
> > drw-------. 2 root root  4096 Feb  1 11:11 checkpoint
> > -rw-------. 1 root root   616 Feb  1 11:11 config.dump
> > -rw-------. 1 root root     0 Feb  1 11:11 dump.log
> > -rw-r--r--. 1 root root   315 Feb  1 11:11 io.kubernetes.cri-o.LogPath
> > -rw-r--r--. 1 root root  2048 Feb  1 11:11 rootfs-diff.tar
> > -rw-------. 1 root root 11276 Feb  1 11:11 spec.dump
> > -rw-r--r--. 1 root root    49 Feb  1 11:11 stats-dump
> >
> > We currently have some metadata defined in
> > github.com/checkpoint-restore/checkpointctl which we want to use in
> > all three projects (podman, containerd and cri-o).
> 
> You know, maybe there's a difference between CRI-O and containerd.
> I followed the steps from the containerd test to create an image:
> https://github.com/containerd/containerd/blob/main/contrib/checkpoint/checkpoint-restore-kubernetes-test.sh#L105
> 
> root@gke-cluster-1-default-pool-595f3f31-2wft:/home/avagin# docker
> create --name test-image avagin/test-cpt:0.5 ls
> c80dbf467d99a0e3a6684d6cb36d29c212a6b12bdfc9af8abe2ffe3fcb69a5de
> root@gke-cluster-1-default-pool-595f3f31-2wft:/home/avagin# docker
> export test-image | tar -t
> .dockerenv
> blobs/
> blobs/sha256/
> blobs/sha256/5159244823d7bfa959a4249c912ffef669c5596fcf41a866264823152b6dbba9
> blobs/sha256/9178f6d56b033b8221dda746c3fd9ad98552569f05e66241365ef8a722da96be
> blobs/sha256/eca4c8bdd20acb007a5594777ace63727d2c17413a54d3a5a817e252d0390902
> dev/
> dev/console
> dev/pts/
> dev/shm/
> etc/
> etc/hostname
> etc/hosts
> etc/mtab
> etc/resolv.conf
> index.json
> oci-layout
> proc/
> sys/
> root@gke-cluster-1-default-pool-595f3f31-2wft:/home/avagin# docker
> export test-image | tar -x -C test-img/
> # tar -tf test-img/blobs/sha256/eca4c8bdd20acb007a5594777ace63727d2c17413a54d3a5a817e252d0390902
> checkpoint/
> checkpoint/cgroup.img
> checkpoint/core-1.img
> checkpoint/core-8.img
> checkpoint/descriptors.json
> checkpoint/fdinfo-2.img
> checkpoint/fdinfo-3.img
> checkpoint/files.img
> checkpoint/fs-1.img

I am a bit confused. Using the following steps I see this:

# kubectl apply -f /root/sleep.yaml
pod/sleeper created
# CP=$(curl -s --insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key -X POST "https://localhost:10250/checkpoint/default/sleeper/sleep"  | jq -r ".items[0]")
# newcontainer=$(buildah from scratch)
# buildah add "$newcontainer" $CP /
# buildah config --annotation=org.criu.checkpoint.container.name=test "$newcontainer"
# buildah commit "$newcontainer" checkpoint-image:latest
# buildah rm "$newcontainer"
# podman image mount checkpoint-image:latest
/var/lib/containers/storage/overlay/58681367751de52d5c779da8ee826d3ba51b21c880e4051f88ee64746d02017e/merged
# ls -la /var/lib/containers/storage/overlay/58681367751de52d5c779da8ee826d3ba51b21c880e4051f88ee64746d02017e/merged
total 32
dr-xr-xr-x. 1 root root  155 Jun 22 13:27 .
drwx------. 6 root root   69 Jun 22 13:27 ..
drwx------. 2 root root 4096 Jun 22 13:27 checkpoint
-rw-------. 1 root root  555 Jun 22 13:27 config.dump
-rw-------. 1 root root    0 Jun 22 13:27 container.log
-rw-r--r--. 1 root root  202 Jun 22 13:27 rootfs-diff.tar
-rw-r--r--. 1 root root 4424 Jun 22 13:27 spec.dump
-rw-------. 1 root root   46 Jun 22 13:27 stats-dump
-rw-------. 1 root root  298 Jun 22 13:27 status
-rw-------. 1 root root 1666 Jun 22 13:27 status.dump
# cat /var/lib/containers/storage/overlay/58681367751de52d5c779da8ee826d3ba51b21c880e4051f88ee64746d02017e/merged/config.dump  | jq
{
  "id": "d974adb0cc366bbb49ef83123eac019f2326b90c5af6eab18db0abb6a084c329",
  "name": "sleep_sleeper_default_250c35ee-e0a4-4bf2-a681-09d7b3faf175_1",
  "rootfsImage": "quay.io/adrianreber/sleep:alpine",
  "rootfsImageRef": "quay.io/adrianreber/sleep@sha256:d504e702fa984e59d0573ff23a16023adb16a5405abf4ba35a64a62dbc9d3a6d",
  "rootfsImageName": "quay.io/adrianreber/sleep:alpine",
  "runtime": "io.containerd.runc.v2",
  "createdTime": "2025-06-22T11:27:13.446696907Z",
  "checkpointedTime": "2025-06-22T13:27:18.835719622+02:00",
  "restoredTime": "0001-01-01T00:00:00Z",
  "restored": false
}

I guess docker export provides something else than podman image mount.

But, whatever we have right now, we can change it to something better.
No problem. We are the authors of all the implementations in containerd
and CRI-O (and Podman) and can change it.

		Adrian