[PATCH 0/1] image-oci: cross-image OCI layer dedup broken by rsync's mtime handling

[PATCH 0/1] image-oci: cross-image OCI layer dedup broken by rsync's mtime handling

[Koen Kooi]

Hi!

While working on the project that triggered "image-oci: drop *-initial
sysroot deps in do_image_oci to avoid libgcc collision", I found
another issue with reusing layers. The timestamp that the reproducible
builds feature locks in doesn't play well with rsync.

The issue:
oci_multilayer_install_packages() merges a layer's freshly installed
rootfs into the already-unpacked OCI_BASE_IMAGE bundle with plain
"rsync -a", which decides whether to re-transfer a file by size+mtime,
not content. SOURCE_DATE_EPOCH is resolved per recipe, so the same
package installed independently for a base image and for an app image
is identical but gets a different mtime each time. -a also implies
--times, so the mismatched re-copy overwrites the base's matching
mtime too. umoci repack then sees a "changed" file and duplicates its
content into the new layer instead of reusing the base's blob by
digest.

It's fairly easy to reproduce[1]:

  # repro-base.bb
  OCI_LAYER_MODE = "multi"
  OCI_LAYERS = "base:packages:coreutils"
  IMAGE_FSTYPES = "container oci"
  inherit image
  inherit image-oci

  # repro-app.bb
  OCI_BASE_IMAGE = "repro-base"
  OCI_LAYER_MODE = "multi"
  OCI_LAYERS = "app:packages:coreutils"
  IMAGE_FSTYPES = "container oci"
  inherit image
  inherit image-oci

Base image manifest: one layer, sha256:5b133f90..., 16 MiB
App image manifest, stock rsync -a: two layers:
  sha256:5b133f90... (same digest, correctly shared) plus a second
  layer, sha256:64f1136e..., a *different* digest at the *same* 16 MiB
Coreutils is completely duplicated under a new digest :(

With "rsync -a --checksum --no-times" in
oci_multilayer_install_packages(): the second layer drops to
sha256:536fe286..., 14 KiB, as expected.

regards,

Koen

[1] This will of course run into the *-initial issue, I still don't
have a proper fix for that.

Koen Kooi (1):
  image-oci: use rsync --checksum --no-times for cross-image layer
    dedup

 classes/image-oci-umoci.inc | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

--
2.34.1

[PATCH 1/1] image-oci: use rsync --checksum --no-times for cross-image layer dedup

[Koen Kooi]

An app image sharing OCI_BASE_IMAGE with a base image should not
duplicate content the base already provides. It does:

- oci_multilayer_install_packages() merges the layer rootfs into the
  base bundle with plain "rsync -a", which only checks mtime+size.
- The per-recipe SOURCE_DATE_EPOCH makes the same recipe build for
  different images have different mtimes, so "rsync -a" wrongly thinks
  the files are different and will re-copy it every time.
- rsync -a also implies --times, so the re-copy overwrites the
  matching mtime the base already had, confirmed via
  --itemize-changes: ">f..t...... usr/lib/libc.so.6".
- umoci repack then sees a changed file and duplicates its content
  into the new layer :(

Add --checksum and --no-times. Reproduced with a base image and an
app image both installing coreutils via a "packages:" layer:
unpatched, the app's own layer duplicates the base's full 16 MiB
content under a new digest; patched, that layer drops to a 14 KiB
residual, as was expected.

Signed-off-by: Koen Kooi <koen.kooi@oss.qualcomm.com>
---
 classes/image-oci-umoci.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/classes/image-oci-umoci.inc b/classes/image-oci-umoci.inc
index a033d73a..67f6c6e5 100644
--- a/classes/image-oci-umoci.inc
+++ b/classes/image-oci-umoci.inc
@@ -582,8 +582,8 @@ IMAGE_CMD:oci() {

                 if [ -d "$oci_preinstall_rootfs" ]; then
                     bbnote "OCI: Copying pre-installed packages from $oci_preinstall_rootfs"
-                    # Use rsync to merge into bundle rootfs (handles symlinks properly)
-                    rsync -a --no-owner --no-group "$oci_preinstall_rootfs/" "$image_bundle_name/rootfs/"
+                    # Use rsync to merge into bundle rootfs (handles symlinks properly), use content, not size+date to deduplicate, SOURCE_DATE_EPOCH breaks mtime.
+                    rsync -a --checksum --no-times --no-owner --no-group "$oci_preinstall_rootfs/" "$image_bundle_name/rootfs/"

                     # Create usrmerge symlinks in the first packages layer.
                     # Normally done by ROOTFS_PREPROCESS_COMMAND in do_rootfs,
--
2.47.3

Re: [meta-virtualization] [PATCH 1/1] image-oci: use rsync --checksum --no-times for cross-image layer dedup

[Bruce Ashfield]

merged

Bruce