[Buildroot] [PATCH next 1/2] package/pkg-cargo: provide a single definition of CARGO_HOME value

[Buildroot] [PATCH next 1/2] package/pkg-cargo: provide a single definition of CARGO_HOME value

[Thomas Petazzoni via buildroot]

Instead of replicating $(HOST_DIR)/share/cargo in several place,
define BR_CARGO_HOME. This will help when we'll want to change this
location.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-cargo.mk         | 6 ++++--
 package/suricata/suricata.mk | 2 +-
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
index f7e3f39503..5923725f5e 100644
--- a/package/pkg-cargo.mk
+++ b/package/pkg-cargo.mk
@@ -20,8 +20,10 @@
 #
 ################################################################################
 
+BR_CARGO_HOME = $(HOST_DIR)/share/cargo
+
 PKG_COMMON_CARGO_ENV = \
-	CARGO_HOME=$(HOST_DIR)/share/cargo
+	CARGO_HOME=$(BR_CARGO_HOME)
 
 # __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS is needed to allow
 # passing the -Z target-applies-to-host, which is needed together with
@@ -79,7 +81,7 @@ $(2)_DOWNLOAD_DEPENDENCIES += host-rustc
 $(2)_DEPENDENCIES += host-rustc
 
 $(2)_DOWNLOAD_POST_PROCESS = cargo
-$(2)_DL_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+$(2)_DL_ENV += CARGO_HOME=$$(BR_CARGO_HOME)
 
 # If building in a sub directory, use that to find the Cargo.toml
 ifneq ($$($(2)_SUBDIR),)
diff --git a/package/suricata/suricata.mk b/package/suricata/suricata.mk
index 8c38b7c713..72b72cc94d 100644
--- a/package/suricata/suricata.mk
+++ b/package/suricata/suricata.mk
@@ -29,7 +29,7 @@ SURICATA_DEPENDENCIES = \
 
 SURICATA_CONF_ENV = \
 	ac_cv_path_HAVE_SPHINXBUILD=no \
-	CARGO_HOME=$(HOST_DIR)/share/cargo \
+	CARGO_HOME=$(BR_CARGO_HOME) \
 	RUST_TARGET=$(RUSTC_TARGET_NAME)
 
 SURICATA_CONF_OPTS = \
-- 
2.38.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Thomas Petazzoni via buildroot]

CARGO_HOME is where Cargo stores its downloaded artefacts. See
https://doc.rust-lang.org/cargo/reference/environment-variables.html:

  CARGO_HOME — Cargo maintains a local cache of the registry index and
  of git checkouts of crates. By default these are stored under
  $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
  overrides the location of this directory. Once a crate is cached it
  is not removed by the clean command. For more details refer to the
  guide.

We currently make it point to $(HOST_DIR)/share/cargo, but this has a
number of drawbacks:

 (1) It is not shared between Buildroot builds. Each Buildroot build
     will re-download the crates index, and the crates themselves,
     unless of course the final vendored tarball is already there.

 (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
     is not even shared between packages, as $(HOST_DIR)/share/cargo
     is per package. So each package in the build that needs vendoring
     of Cargo crates will download the crates index and the crates in
     its own CARGO_HOME location.

To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
is shared between builds and packages.

Even though not the best/most authoritative source,
https://github.com/rust-lang/cargo/issues/6930 indicates that there is
a lock when accessing CARGO_HOME, because a user even complains that
this lock has even become more coarse-grained than it used to be
(which for us is fine, it just means that two Cargo fetch operations
from two different packages will be serialized, not a big deal).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-cargo.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
index 5923725f5e..c3d1e34309 100644
--- a/package/pkg-cargo.mk
+++ b/package/pkg-cargo.mk
@@ -20,7 +20,7 @@
 #
 ################################################################################
 
-BR_CARGO_HOME = $(HOST_DIR)/share/cargo
+BR_CARGO_HOME = $(DL_DIR)/.cargo
 
 PKG_COMMON_CARGO_ENV = \
 	CARGO_HOME=$(BR_CARGO_HOME)
-- 
2.38.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Moritz Bitsch via buildroot]

Am 03.11.22 um 23:45 schrieb Thomas Petazzoni:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
> 
>    CARGO_HOME — Cargo maintains a local cache of the registry index and
>    of git checkouts of crates. By default these are stored under
>    $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
>    overrides the location of this directory. Once a crate is cached it
>    is not removed by the clean command. For more details refer to the
>    guide.
> 
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
> 
>   (1) It is not shared between Buildroot builds. Each Buildroot build
>       will re-download the crates index, and the crates themselves,
>       unless of course the final vendored tarball is already there.
> 
>   (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
>       is not even shared between packages, as $(HOST_DIR)/share/cargo
>       is per package. So each package in the build that needs vendoring
>       of Cargo crates will download the crates index and the crates in
>       its own CARGO_HOME location.
> 
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.
> 
> Even though not the best/most authoritative source,
> https://github.com/rust-lang/cargo/issues/6930 indicates that there is
> a lock when accessing CARGO_HOME, because a user even complains that
> this lock has even become more coarse-grained than it used to be
> (which for us is fine, it just means that two Cargo fetch operations
> from two different packages will be serialized, not a big deal).

For my builds/testcases the shared cache improves downloads, even with 
the locking. (Tested with gigabit connection and -j 24 on a fast NVMe 
system)

Tested-by: Moritz Bitsch <moritz@h6t.eu>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>   package/pkg-cargo.mk | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> index 5923725f5e..c3d1e34309 100644
> --- a/package/pkg-cargo.mk
> +++ b/package/pkg-cargo.mk
> @@ -20,7 +20,7 @@
>   #
>   ################################################################################
>   
> -BR_CARGO_HOME = $(HOST_DIR)/share/cargo
> +BR_CARGO_HOME = $(DL_DIR)/.cargo
>   
>   PKG_COMMON_CARGO_ENV = \
>   	CARGO_HOME=$(BR_CARGO_HOME)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Yann E. MORIN]

Thomas, All,

On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
> 
>   CARGO_HOME — Cargo maintains a local cache of the registry index and
>   of git checkouts of crates. By default these are stored under
>   $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
>   overrides the location of this directory. Once a crate is cached it
>   is not removed by the clean command. For more details refer to the
>   guide.
> 
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
> 
>  (1) It is not shared between Buildroot builds. Each Buildroot build
>      will re-download the crates index, and the crates themselves,
>      unless of course the final vendored tarball is already there.
> 
>  (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
>      is not even shared between packages, as $(HOST_DIR)/share/cargo
>      is per package. So each package in the build that needs vendoring
>      of Cargo crates will download the crates index and the crates in
>      its own CARGO_HOME location.
> 
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.
> 
> Even though not the best/most authoritative source,
> https://github.com/rust-lang/cargo/issues/6930 indicates that there is
> a lock when accessing CARGO_HOME, because a user even complains that
> this lock has even become more coarse-grained than it used to be
> (which for us is fine, it just means that two Cargo fetch operations
> from two different packages will be serialized, not a big deal).
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-cargo.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> index 5923725f5e..c3d1e34309 100644
> --- a/package/pkg-cargo.mk
> +++ b/package/pkg-cargo.mk
> @@ -20,7 +20,7 @@
>  #
>  ################################################################################
>  
> -BR_CARGO_HOME = $(HOST_DIR)/share/cargo
> +BR_CARGO_HOME = $(DL_DIR)/.cargo

I am not a fan of dot-directories, because they get hidden, especially
since BR2_DL_DIR can be set in the environment and point to a long-lived
directory.

Instead, I would just call it 'cargo-home'. If that's OK, we can change
when applying...

Regards,
Yann E. MORIN.

>  PKG_COMMON_CARGO_ENV = \
>  	CARGO_HOME=$(BR_CARGO_HOME)
> -- 
> 2.38.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 1/2] package/pkg-cargo: provide a single definition of CARGO_HOME value

[Yann E. MORIN]

Thomas, All,

On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> Instead of replicating $(HOST_DIR)/share/cargo in several place,
> define BR_CARGO_HOME. This will help when we'll want to change this
> location.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Applied to next, thanks.

Regards,
Yann E. MORIN.

> ---
>  package/pkg-cargo.mk         | 6 ++++--
>  package/suricata/suricata.mk | 2 +-
>  2 files changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> index f7e3f39503..5923725f5e 100644
> --- a/package/pkg-cargo.mk
> +++ b/package/pkg-cargo.mk
> @@ -20,8 +20,10 @@
>  #
>  ################################################################################
>  
> +BR_CARGO_HOME = $(HOST_DIR)/share/cargo
> +
>  PKG_COMMON_CARGO_ENV = \
> -	CARGO_HOME=$(HOST_DIR)/share/cargo
> +	CARGO_HOME=$(BR_CARGO_HOME)
>  
>  # __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS is needed to allow
>  # passing the -Z target-applies-to-host, which is needed together with
> @@ -79,7 +81,7 @@ $(2)_DOWNLOAD_DEPENDENCIES += host-rustc
>  $(2)_DEPENDENCIES += host-rustc
>  
>  $(2)_DOWNLOAD_POST_PROCESS = cargo
> -$(2)_DL_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
> +$(2)_DL_ENV += CARGO_HOME=$$(BR_CARGO_HOME)
>  
>  # If building in a sub directory, use that to find the Cargo.toml
>  ifneq ($$($(2)_SUBDIR),)
> diff --git a/package/suricata/suricata.mk b/package/suricata/suricata.mk
> index 8c38b7c713..72b72cc94d 100644
> --- a/package/suricata/suricata.mk
> +++ b/package/suricata/suricata.mk
> @@ -29,7 +29,7 @@ SURICATA_DEPENDENCIES = \
>  
>  SURICATA_CONF_ENV = \
>  	ac_cv_path_HAVE_SPHINXBUILD=no \
> -	CARGO_HOME=$(HOST_DIR)/share/cargo \
> +	CARGO_HOME=$(BR_CARGO_HOME) \
>  	RUST_TARGET=$(RUSTC_TARGET_NAME)
>  
>  SURICATA_CONF_OPTS = \
> -- 
> 2.38.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Thomas Petazzoni via buildroot]

Hello,

On Sun, 6 Nov 2022 16:03:30 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> I am not a fan of dot-directories, because they get hidden, especially
> since BR2_DL_DIR can be set in the environment and point to a long-lived
> directory.
> 
> Instead, I would just call it 'cargo-home'. If that's OK, we can change
> when applying...

I don't feel strongly about it. I used .cargo because that's how the
default CARGO_HOME folder is named in one's $HOME directory. Also, it
makes sure there is no potential conflict with an hypothetical
cargo-home package. But overall, I agree that I also don't like it
being hidden. We could also name it br-cargo-home, so that it doesn't
really look like a package download directory.

But really, I don't feel strongly about it and $(DL_DIR)/cargo-home
works fine for me.

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Yann E. MORIN]

Thomas, All,

On 2022-11-07 08:52 +0100, Thomas Petazzoni spake thusly:
> On Sun, 6 Nov 2022 16:03:30 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > I am not a fan of dot-directories, because they get hidden, especially
> > since BR2_DL_DIR can be set in the environment and point to a long-lived
> > directory.
> > 
> > Instead, I would just call it 'cargo-home'. If that's OK, we can change
> > when applying...
> 
> I don't feel strongly about it. I used .cargo because that's how the
> default CARGO_HOME folder is named in one's $HOME directory. Also, it
> makes sure there is no potential conflict with an hypothetical
> cargo-home package. But overall, I agree that I also don't like it
> being hidden. We could also name it br-cargo-home, so that it doesn't
> really look like a package download directory.
> 
> But really, I don't feel strongly about it and $(DL_DIR)/cargo-home
> works fine for me.

I liked the br-cargo-home, so that's what I renamed it to.

Applied to next, thanks.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Yann E. MORIN]

Thomas, All,

On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
> 
>   CARGO_HOME — Cargo maintains a local cache of the registry index and
>   of git checkouts of crates. By default these are stored under
>   $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
>   overrides the location of this directory. Once a crate is cached it
>   is not removed by the clean command. For more details refer to the
>   guide.
> 
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
> 
>  (1) It is not shared between Buildroot builds. Each Buildroot build
>      will re-download the crates index, and the crates themselves,
>      unless of course the final vendored tarball is already there.
> 
>  (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
>      is not even shared between packages, as $(HOST_DIR)/share/cargo
>      is per package. So each package in the build that needs vendoring
>      of Cargo crates will download the crates index and the crates in
>      its own CARGO_HOME location.
> 
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.

Unfortunately, this causes breakage when there are parallel downloads
(see below for except; full log as attachment).

So, there are two solutions:

 1. revert to a per-package cargo home, losing the benefit of a shared
    cache, but trivial to implement

 2. add locking ourselves in the post-process scripts, so that there are
    never two "cargo vendor" running in parallel, which is slightly more
    involved.

 3. tell upstream that they borked their locking?

Toughts?

Note: doing sequential downloads works, of course...

>>> dust 0.8.1 Downloading
wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output' 'https://github.com/bootandy/dust/archive/v0.8.1/dust-0.8.1.tar.gz'
[--SNIP--]
2023-01-14 14:30:15 (1.42 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output’ saved [89023/89023]

    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
 Downloading crates ...
error: failed to sync

Caused by:
  failed to download packages

Caused by:
  failed to download `autocfg v1.1.0`

Caused by:
  unable to get packages from source

Caused by:
  failed to unpack package `autocfg v1.1.0`

Caused by:
  failed to unpack entry at `autocfg-1.1.0/src/tests.rs`

Caused by:
  No such file or directory (os error 2) while canonicalizing /home/ymorin/dev/buildroot/O/master/yem-dl/br-cargo-home/registry/src/github.com-1ecc6299db9ec823/autocfg-1.1.0/src
make[1]: *** [package/pkg-generic.mk:189: /home/ymorin/dev/buildroot/O/master/build/dust-0.8.1/.stamp_downloaded] Error 101
make[1]: *** Waiting for unfinished jobs....

>>> ripgrep af6b6c543b224d348a8876f0c06245d9ea7929c5 Downloading
wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/output' 'https://github.com/burntsushi/ripgrep/archive/af6b6c543b224d348a8876f0c06245d9ea7929c5/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz'
[--SNIP--]
2023-01-14 14:30:16 (1.69 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/output’ saved [506803/506803]

warning: Both `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config` and `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config.toml` exist. Using `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
error: failed to sync

Caused by:
  failed to download packages

Caused by:
  failed to download `aho-corasick v0.7.18`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/ymorin/dev/buildroot/O/master/yem-dl/br-cargo-home/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18/Cargo.toml`

Caused by:
  can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path
make[1]: *** [package/pkg-generic.mk:189: /home/ymorin/dev/buildroot/O/master/build/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.stamp_downloaded] Error 101

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Thomas Petazzoni via buildroot]

On Sat, 14 Jan 2023 14:58:17 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Unfortunately, this causes breakage when there are parallel downloads
> (see below for except; full log as attachment).

Thanks for the report.

> 
> So, there are two solutions:
> 
>  1. revert to a per-package cargo home, losing the benefit of a shared
>     cache, but trivial to implement
> 
>  2. add locking ourselves in the post-process scripts, so that there are
>     never two "cargo vendor" running in parallel, which is slightly more
>     involved.

Is that really that involved?

>  3. tell upstream that they borked their locking?

We should obviously tell them.

> >>> dust 0.8.1 Downloading  
> wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output' 'https://github.com/bootandy/dust/archive/v0.8.1/dust-0.8.1.tar.gz'
> [--SNIP--]
> 2023-01-14 14:30:15 (1.42 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output’ saved [89023/89023]
> 
>     Blocking waiting for file lock on package cache
>     Blocking waiting for file lock on package cache

So they do have locking but it doesn't work?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR

[Yann E. MORIN]

Thomas, All,

On 2023-01-14 14:58 +0100, Yann E. MORIN spake thusly:
> On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> > CARGO_HOME is where Cargo stores its downloaded artefacts. See
> > https://doc.rust-lang.org/cargo/reference/environment-variables.html:
[--SNIP--]
> > To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> > is shared between builds and packages.
> Unfortunately, this causes breakage when there are parallel downloads
> (see below for except; full log as attachment).
> So, there are two solutions:
>  1. revert to a per-package cargo home, losing the benefit of a shared
>     cache, but trivial to implement
>  2. add locking ourselves in the post-process scripts, so that there are
>     never two "cargo vendor" running in parallel, which is slightly more
>     involved.

In fact, it turns out that doing the lock ourselves is relatively
trivial, so I'll do a few more tests and send a patch...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot