[PATCH 0/3] Use frozen flag for cargo build

[PATCH 0/3] Use frozen flag for cargo build

[frederic.martinsons]

From: Frederic Martinsons <frederic.martinsons@gmail.com>

This patch series force the usage of --frozen instead of --offline during cargo build.
This has the advantage to be sure that Cargo.lock file will not be modified.
Moreover, raise a clear error when Cargo.lock is not present.
For this to work, we must modify ourself the Cargo.lock before building
in case there are git repo that have been patched.

The last commit is to handle the specific case of rust-hello-world recipe.
If this recipe is no more necessary, we can drop this last patch.

The following changes since commit 9b5b850d6a6982bb8ff14dcfbb6769b293638293:

  libarchive: ignore CVE-2023-30571 (2023-07-30 12:00:13 +0100)

are available in the Git repository at:

  https://gitlab.com/fmartinsons/openembedded-core 15104-check-cargo-lock

Frederic Martinsons (3):
  cargo.bbclass: Use --frozen flag for cargo operations
  cargo_common.bbclass: Handle Cargo.lock modifications for git
    dependencies
  cargo.bbclass: Offer a way to use --offline instead of --frozen

 meta/classes-recipe/cargo.bbclass             |  9 +++-
 meta/classes-recipe/cargo_common.bbclass      | 43 +++++++++++++++++++
 .../rust-example/rust-hello-world_git.bb      |  2 +
 3 files changed, 53 insertions(+), 1 deletion(-)

-- 
2.34.1

[PATCH 0/3] Use frozen flag for cargo build

[frederic.martinsons]

From: Frederic Martinsons <frederic.martinsons@gmail.com>

This patch series force the usage of --frozen instead of --offline during cargo build.
This has the advantage to be sure that Cargo.lock file will not be modified.
Moreover, raise a clear error when Cargo.lock is not present.
For this to work, we must modify ourself the Cargo.lock before building
in case there are git repo that have been patched.

The last commit was for removing rust-hello-world, patch to the
documentation project will follow.

The following changes since commit 692e414aed5313ff275b69e93179aa7c559700f3:

  ffmpeg: Fix wrong code found with gas/2.41 (2023-08-01 09:49:14 +0100)

are available in the Git repository at:

  https://gitlab.com/fmartinsons/openembedded-core 15104-check-cargo-lock

Frederic Martinsons (3):
  cargo.bbclass: Use --frozen flag for cargo operations
  cargo_common.bbclass: Handle Cargo.lock modifications for git
    dependencies
  drop rust-hello-world recipe

 meta/classes-recipe/cargo.bbclass             |  7 +++-
 meta/classes-recipe/cargo_common.bbclass      | 41 +++++++++++++++++++
 meta/conf/distro/include/maintainers.inc      |  1 -
 meta/lib/oeqa/runtime/cases/rust.py           | 12 ------
 .../packagegroup-core-tools-testapps.bb       |  3 --
 .../rust-hello-world/0001-enable-LTO.patch    | 24 -----------
 .../rust-example/rust-hello-world_git.bb      | 19 ---------
 7 files changed, 47 insertions(+), 60 deletions(-)
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world_git.bb

-- 
2.34.1

[PATCH 1/3] cargo.bbclass: Use --frozen flag for cargo operations

[frederic.martinsons]

From: Frederic Martinsons <frederic.martinsons@gmail.com>

It supersed the --offline flag and guarantee that Cargo.lock
file will not be modified during the build.

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
---
 meta/classes-recipe/cargo.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/cargo.bbclass b/meta/classes-recipe/cargo.bbclass
index 3ef0bbbb44..8c0b92df8d 100644
--- a/meta/classes-recipe/cargo.bbclass
+++ b/meta/classes-recipe/cargo.bbclass
@@ -39,7 +39,12 @@ MANIFEST_PATH ??= "${S}/${CARGO_SRC_DIR}/Cargo.toml"
 
 RUSTFLAGS ??= ""
 BUILD_MODE = "${@['--release', ''][d.getVar('DEBUG_BUILD') == '1']}"
-CARGO_BUILD_FLAGS = "-v --offline --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}"
+# --frozen flag will prevent network access (which is required since only
+# the do_fetch step is authorized to access network)
+# and will require an up to date Cargo.lock file.
+# This force the package being built to already ship a Cargo.lock, in the end
+# this is what we want, at least, for reproducibility of the build.
+CARGO_BUILD_FLAGS = "-v --frozen --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}"
 
 # This is based on the content of CARGO_BUILD_FLAGS and generally will need to
 # change if CARGO_BUILD_FLAGS changes.
-- 
2.34.1

[PATCH 2/3] cargo_common.bbclass: Handle Cargo.lock modifications for git dependencies

[frederic.martinsons]

From: Frederic Martinsons <frederic.martinsons@gmail.com>

Now we use --frozen, Cargo.lock cannot be modified by cargo build.
These patched git dependencies requires that the git url is removed
from Cargo.lock.

Fixes #15104

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
---
 meta/classes-recipe/cargo_common.bbclass | 41 ++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
index db54826ddb..b732a1bd95 100644
--- a/meta/classes-recipe/cargo_common.bbclass
+++ b/meta/classes-recipe/cargo_common.bbclass
@@ -117,6 +117,8 @@ cargo_common_do_configure () {
 }
 
 python cargo_common_do_patch_paths() {
+    import shutil
+
     cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
     if not os.path.exists(cargo_config):
         return
@@ -146,6 +148,45 @@ python cargo_common_do_patch_paths() {
             print('\n[patch."%s"]' % k, file=config)
             for name in v:
                 print(name, file=config)
+
+    if not patches:
+        return
+
+    # Cargo.lock file is needed for to be sure that artifacts
+    # downloaded by the fetch steps are those expected by the
+    # project and that the possible patches are correctly applied.
+    # Moreover since we do not want any modification
+    # of this file (for reproducibility purpose), we prevent it by
+    # using --frozen flag (in CARGO_BUILD_FLAGS) and raise a clear error
+    # here is better than letting cargo tell (in case the file is missing)
+    # "Cargo.lock should be modified but --frozen was given"
+
+    manifest_path = d.getVar("MANIFEST_PATH", True)
+    lockfile = os.path.join(os.path.dirname(manifest_path), "Cargo.lock")
+    if not os.path.exists(lockfile):
+        bb.fatal(f"{lockfile} file doesn't exist")
+
+    # There are patched files and so Cargo.lock should be modified but we use
+    # --frozen so let's handle that modifications here.
+    #
+    # Note that a "better" (more elegant ?) would have been to use cargo update for
+    # patched packages:
+    #  cargo update --offline -p package_1 -p package_2
+    # But this is not possible since it requires that cargo local git db
+    # to be populated and this is not the case as we fetch git repo ourself.
+
+    lockfile_orig = lockfile + ".orig"
+    if not os.path.exists(lockfile_orig):
+        shutil.copy(lockfile, lockfile_orig)
+
+    newlines = []
+    with open(lockfile_orig, "r") as f:
+        for line in f.readlines():
+            if not line.startswith("source = \"git"):
+                newlines.append(line)
+
+    with open(lockfile, "w") as f:
+        f.writelines(newlines)
 }
 do_configure[postfuncs] += "cargo_common_do_patch_paths"
 
-- 
2.34.1

[PATCH 3/3] drop rust-hello-world recipe

[frederic.martinsons]

From: Frederic Martinsons <frederic.martinsons@gmail.com>

we now have more rust/cargo recipes and test that covers
various use cases.

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
---
 meta/conf/distro/include/maintainers.inc      |  1 -
 meta/lib/oeqa/runtime/cases/rust.py           | 12 ----------
 .../packagegroup-core-tools-testapps.bb       |  3 ---
 .../rust-hello-world/0001-enable-LTO.patch    | 24 -------------------
 .../rust-example/rust-hello-world_git.bb      | 19 ---------------
 5 files changed, 59 deletions(-)
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world_git.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index 1c9c5616d8..202e5739a0 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -734,7 +734,6 @@ RECIPE_MAINTAINER:pn-ruby = "Ross Burton <ross.burton@arm.com>"
 RECIPE_MAINTAINER:pn-run-postinsts = "Ross Burton <ross.burton@arm.com>"
 RECIPE_MAINTAINER:pn-rust = "Randy MacLeod <Randy.MacLeod@windriver.com>"
 RECIPE_MAINTAINER:pn-rust-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Randy MacLeod <Randy.MacLeod@windriver.com>"
-RECIPE_MAINTAINER:pn-rust-hello-world = "Randy MacLeod <Randy.MacLeod@windriver.com>"
 RECIPE_MAINTAINER:pn-rust-llvm = "Randy MacLeod <Randy.MacLeod@windriver.com>"
 RECIPE_MAINTAINER:pn-rxvt-unicode = "Unassigned <unassigned@yoctoproject.org>"
 RECIPE_MAINTAINER:pn-sato-screenshot = "Ross Burton <ross.burton@arm.com>"
diff --git a/meta/lib/oeqa/runtime/cases/rust.py b/meta/lib/oeqa/runtime/cases/rust.py
index c9c60e16fd..9bf0312669 100644
--- a/meta/lib/oeqa/runtime/cases/rust.py
+++ b/meta/lib/oeqa/runtime/cases/rust.py
@@ -48,15 +48,3 @@ class RustCompileTest(OERuntimeTestCase):
         status, output = self.target.run('cargo run --manifest-path=/tmp/hello/Cargo.toml')
         msg = 'running compiled file failed, output: %s' % output
         self.assertEqual(status, 0, msg=msg)
-
-class RustHelloworldTest(OERuntimeTestCase):
-    @OETestDepends(['ssh.SSHTest.test_ssh'])
-    @OEHasPackage(['rust-hello-world'])
-    def test_rusthelloworld(self):
-        cmd = "rust-hello-world"
-        status, output = self.target.run(cmd)
-        msg = 'Exit status was not 0. Output: %s' % output
-        self.assertEqual(status, 0, msg=msg)
-
-        msg = 'Incorrect output: %s' % output
-        self.assertEqual(output, "Hello, world!", msg=msg)
diff --git a/meta/recipes-core/packagegroups/packagegroup-core-tools-testapps.bb b/meta/recipes-core/packagegroups/packagegroup-core-tools-testapps.bb
index e05e329020..1fee1c925d 100644
--- a/meta/recipes-core/packagegroups/packagegroup-core-tools-testapps.bb
+++ b/meta/recipes-core/packagegroups/packagegroup-core-tools-testapps.bb
@@ -25,8 +25,6 @@ GOTOOLS ?= "go-helloworld"
 GOTOOLS:powerpc ?= ""
 GOTOOLS:riscv32 ?= ""
 
-RUSTTOOLS ?= "rust-hello-world"
-
 GSTEXAMPLES ?= "gst-examples"
 GSTEXAMPLES:riscv64 = ""
 
@@ -59,5 +57,4 @@ RDEPENDS:${PN} = "\
     ${@bb.utils.contains('DISTRO_FEATURES', 'x11 opengl', "${X11GLTOOLS}", "", d)} \
     ${@bb.utils.contains('DISTRO_FEATURES', '3g', "${3GTOOLS}", "", d)} \
     ${GOTOOLS} \
-    ${RUSTTOOLS} \
     "
diff --git a/meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch b/meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
deleted file mode 100644
index f319545ee1..0000000000
--- a/meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
+++ /dev/null
@@ -1,24 +0,0 @@
-From fa40b874f6470ec11a8fd7b0c9909d0cdd2d6feb Mon Sep 17 00:00:00 2001
-From: Dan Callaghan <dan.callaghan@opengear.com>
-Date: Fri, 5 Feb 2021 08:56:34 +1000
-Subject: [PATCH] enable LTO
-
-Upstream-Status: Pending
----
- Cargo.toml | 3 +++
- 1 file changed, 3 insertions(+)
-
-diff --git a/Cargo.toml b/Cargo.toml
-index 7a2f6c8..cdb6b5d 100644
---- a/Cargo.toml
-+++ b/Cargo.toml
-@@ -3,3 +3,6 @@
- name = "rust-hello-world"
- version = "0.0.1"
- authors = ["Cody P Schafer <dev@codyps.com>"]
-+
-+[profile.release]
-+lto = true
--- 
-2.28.0
-
diff --git a/meta/recipes-extended/rust-example/rust-hello-world_git.bb b/meta/recipes-extended/rust-example/rust-hello-world_git.bb
deleted file mode 100644
index 1d91109b51..0000000000
--- a/meta/recipes-extended/rust-example/rust-hello-world_git.bb
+++ /dev/null
@@ -1,19 +0,0 @@
-inherit cargo
-
-SRC_URI = "git://github.com/meta-rust/rust-hello-world.git;protocol=https;branch=master"
-SRCREV="e0fa23f1a3cb1eb1407165bd2fc36d2f6e6ad728"
-LIC_FILES_CHKSUM="file://COPYRIGHT;md5=e6b2207ac3740d2d01141c49208c2147"
-
-SRC_URI += "\
-    file://0001-enable-LTO.patch \
-    "
-
-UPSTREAM_CHECK_COMMITS = "1"
-
-SUMMARY = "Hello World by Cargo for Rust"
-HOMEPAGE = "https://github.com/meta-rust/rust-hello-world"
-LICENSE = "MIT | Apache-2.0"
-
-S = "${WORKDIR}/git"
-
-BBCLASSEXTEND = "native"
-- 
2.34.1

Re: [PATCH 0/3] Use frozen flag for cargo build

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 2022 bytes --]

Le mer. 2 août 2023, 15:16, <frederic.martinsons@gmail.com> a écrit :

> From: Frederic Martinsons <frederic.martinsons@gmail.com>
>
> This patch series force the usage of --frozen instead of --offline during
> cargo build.
> This has the advantage to be sure that Cargo.lock file will not be
> modified.
> Moreover, raise a clear error when Cargo.lock is not present.
> For this to work, we must modify ourself the Cargo.lock before building
> in case there are git repo that have been patched.
>
> The last commit was for removing rust-hello-world, patch to the
> documentation project will follow.
>
> The following changes since commit
> 692e414aed5313ff275b69e93179aa7c559700f3:
>
>   ffmpeg: Fix wrong code found with gas/2.41 (2023-08-01 09:49:14 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/fmartinsons/openembedded-core 15104-check-cargo-lock
>
> Frederic Martinsons (3):
>   cargo.bbclass: Use --frozen flag for cargo operations
>   cargo_common.bbclass: Handle Cargo.lock modifications for git
>     dependencies
>   drop rust-hello-world recipe
>
>  meta/classes-recipe/cargo.bbclass             |  7 +++-
>  meta/classes-recipe/cargo_common.bbclass      | 41 +++++++++++++++++++
>  meta/conf/distro/include/maintainers.inc      |  1 -
>  meta/lib/oeqa/runtime/cases/rust.py           | 12 ------
>  .../packagegroup-core-tools-testapps.bb       |  3 --
>  .../rust-hello-world/0001-enable-LTO.patch    | 24 -----------
>  .../rust-example/rust-hello-world_git.bb      | 19 ---------
>  7 files changed, 47 insertions(+), 60 deletions(-)
>  delete mode 100644
> meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
>  delete mode 100644 meta/recipes-extended/rust-example/
> rust-hello-world_git.bb
>
> --
> 2.34.1
>

Sorry, I missed the v2 part in header. These are the follow up after
remarks made yesterday (about removal of rust-hello-world and robustness
when modifying Cargo.lock file in place)

>

[-- Attachment #2: Type: text/html, Size: 3106 bytes --]