* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2022-11-03 22:45 ` [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR Thomas Petazzoni via buildroot
@ 2022-11-04 8:54 ` Moritz Bitsch via buildroot
2022-11-06 15:03 ` Yann E. MORIN
2023-01-14 13:58 ` Yann E. MORIN
2 siblings, 0 replies; 10+ messages in thread
From: Moritz Bitsch via buildroot @ 2022-11-04 8:54 UTC (permalink / raw)
To: Thomas Petazzoni, Buildroot List; +Cc: James Hilliard
Am 03.11.22 um 23:45 schrieb Thomas Petazzoni:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
>
> CARGO_HOME — Cargo maintains a local cache of the registry index and
> of git checkouts of crates. By default these are stored under
> $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
> overrides the location of this directory. Once a crate is cached it
> is not removed by the clean command. For more details refer to the
> guide.
>
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
>
> (1) It is not shared between Buildroot builds. Each Buildroot build
> will re-download the crates index, and the crates themselves,
> unless of course the final vendored tarball is already there.
>
> (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
> is not even shared between packages, as $(HOST_DIR)/share/cargo
> is per package. So each package in the build that needs vendoring
> of Cargo crates will download the crates index and the crates in
> its own CARGO_HOME location.
>
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.
>
> Even though not the best/most authoritative source,
> https://github.com/rust-lang/cargo/issues/6930 indicates that there is
> a lock when accessing CARGO_HOME, because a user even complains that
> this lock has even become more coarse-grained than it used to be
> (which for us is fine, it just means that two Cargo fetch operations
> from two different packages will be serialized, not a big deal).
For my builds/testcases the shared cache improves downloads, even with
the locking. (Tested with gigabit connection and -j 24 on a fast NVMe
system)
Tested-by: Moritz Bitsch <moritz@h6t.eu>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> package/pkg-cargo.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> index 5923725f5e..c3d1e34309 100644
> --- a/package/pkg-cargo.mk
> +++ b/package/pkg-cargo.mk
> @@ -20,7 +20,7 @@
> #
> ################################################################################
>
> -BR_CARGO_HOME = $(HOST_DIR)/share/cargo
> +BR_CARGO_HOME = $(DL_DIR)/.cargo
>
> PKG_COMMON_CARGO_ENV = \
> CARGO_HOME=$(BR_CARGO_HOME)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2022-11-03 22:45 ` [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR Thomas Petazzoni via buildroot
2022-11-04 8:54 ` Moritz Bitsch via buildroot
@ 2022-11-06 15:03 ` Yann E. MORIN
2022-11-07 7:52 ` Thomas Petazzoni via buildroot
2023-01-14 13:58 ` Yann E. MORIN
2 siblings, 1 reply; 10+ messages in thread
From: Yann E. MORIN @ 2022-11-06 15:03 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: Moritz Bitsch, James Hilliard, Buildroot List
Thomas, All,
On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
>
> CARGO_HOME — Cargo maintains a local cache of the registry index and
> of git checkouts of crates. By default these are stored under
> $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
> overrides the location of this directory. Once a crate is cached it
> is not removed by the clean command. For more details refer to the
> guide.
>
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
>
> (1) It is not shared between Buildroot builds. Each Buildroot build
> will re-download the crates index, and the crates themselves,
> unless of course the final vendored tarball is already there.
>
> (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
> is not even shared between packages, as $(HOST_DIR)/share/cargo
> is per package. So each package in the build that needs vendoring
> of Cargo crates will download the crates index and the crates in
> its own CARGO_HOME location.
>
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.
>
> Even though not the best/most authoritative source,
> https://github.com/rust-lang/cargo/issues/6930 indicates that there is
> a lock when accessing CARGO_HOME, because a user even complains that
> this lock has even become more coarse-grained than it used to be
> (which for us is fine, it just means that two Cargo fetch operations
> from two different packages will be serialized, not a big deal).
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> package/pkg-cargo.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> index 5923725f5e..c3d1e34309 100644
> --- a/package/pkg-cargo.mk
> +++ b/package/pkg-cargo.mk
> @@ -20,7 +20,7 @@
> #
> ################################################################################
>
> -BR_CARGO_HOME = $(HOST_DIR)/share/cargo
> +BR_CARGO_HOME = $(DL_DIR)/.cargo
I am not a fan of dot-directories, because they get hidden, especially
since BR2_DL_DIR can be set in the environment and point to a long-lived
directory.
Instead, I would just call it 'cargo-home'. If that's OK, we can change
when applying...
Regards,
Yann E. MORIN.
> PKG_COMMON_CARGO_ENV = \
> CARGO_HOME=$(BR_CARGO_HOME)
> --
> 2.38.1
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2022-11-06 15:03 ` Yann E. MORIN
@ 2022-11-07 7:52 ` Thomas Petazzoni via buildroot
2022-11-07 17:33 ` Yann E. MORIN
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-11-07 7:52 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Moritz Bitsch, James Hilliard, Buildroot List
Hello,
On Sun, 6 Nov 2022 16:03:30 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> I am not a fan of dot-directories, because they get hidden, especially
> since BR2_DL_DIR can be set in the environment and point to a long-lived
> directory.
>
> Instead, I would just call it 'cargo-home'. If that's OK, we can change
> when applying...
I don't feel strongly about it. I used .cargo because that's how the
default CARGO_HOME folder is named in one's $HOME directory. Also, it
makes sure there is no potential conflict with an hypothetical
cargo-home package. But overall, I agree that I also don't like it
being hidden. We could also name it br-cargo-home, so that it doesn't
really look like a package download directory.
But really, I don't feel strongly about it and $(DL_DIR)/cargo-home
works fine for me.
Thanks!
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2022-11-07 7:52 ` Thomas Petazzoni via buildroot
@ 2022-11-07 17:33 ` Yann E. MORIN
0 siblings, 0 replies; 10+ messages in thread
From: Yann E. MORIN @ 2022-11-07 17:33 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: Moritz Bitsch, James Hilliard, Buildroot List
Thomas, All,
On 2022-11-07 08:52 +0100, Thomas Petazzoni spake thusly:
> On Sun, 6 Nov 2022 16:03:30 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > I am not a fan of dot-directories, because they get hidden, especially
> > since BR2_DL_DIR can be set in the environment and point to a long-lived
> > directory.
> >
> > Instead, I would just call it 'cargo-home'. If that's OK, we can change
> > when applying...
>
> I don't feel strongly about it. I used .cargo because that's how the
> default CARGO_HOME folder is named in one's $HOME directory. Also, it
> makes sure there is no potential conflict with an hypothetical
> cargo-home package. But overall, I agree that I also don't like it
> being hidden. We could also name it br-cargo-home, so that it doesn't
> really look like a package download directory.
>
> But really, I don't feel strongly about it and $(DL_DIR)/cargo-home
> works fine for me.
I liked the br-cargo-home, so that's what I renamed it to.
Applied to next, thanks.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2022-11-03 22:45 ` [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR Thomas Petazzoni via buildroot
2022-11-04 8:54 ` Moritz Bitsch via buildroot
2022-11-06 15:03 ` Yann E. MORIN
@ 2023-01-14 13:58 ` Yann E. MORIN
2023-01-14 14:17 ` Thomas Petazzoni via buildroot
2023-01-14 14:19 ` Yann E. MORIN
2 siblings, 2 replies; 10+ messages in thread
From: Yann E. MORIN @ 2023-01-14 13:58 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: Moritz Bitsch, James Hilliard, Buildroot List
Thomas, All,
On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> CARGO_HOME is where Cargo stores its downloaded artefacts. See
> https://doc.rust-lang.org/cargo/reference/environment-variables.html:
>
> CARGO_HOME — Cargo maintains a local cache of the registry index and
> of git checkouts of crates. By default these are stored under
> $HOME/.cargo (%USERPROFILE%\.cargo on Windows), but this variable
> overrides the location of this directory. Once a crate is cached it
> is not removed by the clean command. For more details refer to the
> guide.
>
> We currently make it point to $(HOST_DIR)/share/cargo, but this has a
> number of drawbacks:
>
> (1) It is not shared between Buildroot builds. Each Buildroot build
> will re-download the crates index, and the crates themselves,
> unless of course the final vendored tarball is already there.
>
> (2) With BR2_PER_PACKAGE_DIRECTORIES=y, it is even worse: CARGO_HOME
> is not even shared between packages, as $(HOST_DIR)/share/cargo
> is per package. So each package in the build that needs vendoring
> of Cargo crates will download the crates index and the crates in
> its own CARGO_HOME location.
>
> To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> is shared between builds and packages.
Unfortunately, this causes breakage when there are parallel downloads
(see below for except; full log as attachment).
So, there are two solutions:
1. revert to a per-package cargo home, losing the benefit of a shared
cache, but trivial to implement
2. add locking ourselves in the post-process scripts, so that there are
never two "cargo vendor" running in parallel, which is slightly more
involved.
3. tell upstream that they borked their locking?
Toughts?
Note: doing sequential downloads works, of course...
>>> dust 0.8.1 Downloading
wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output' 'https://github.com/bootandy/dust/archive/v0.8.1/dust-0.8.1.tar.gz'
[--SNIP--]
2023-01-14 14:30:15 (1.42 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output’ saved [89023/89023]
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Downloading crates ...
error: failed to sync
Caused by:
failed to download packages
Caused by:
failed to download `autocfg v1.1.0`
Caused by:
unable to get packages from source
Caused by:
failed to unpack package `autocfg v1.1.0`
Caused by:
failed to unpack entry at `autocfg-1.1.0/src/tests.rs`
Caused by:
No such file or directory (os error 2) while canonicalizing /home/ymorin/dev/buildroot/O/master/yem-dl/br-cargo-home/registry/src/github.com-1ecc6299db9ec823/autocfg-1.1.0/src
make[1]: *** [package/pkg-generic.mk:189: /home/ymorin/dev/buildroot/O/master/build/dust-0.8.1/.stamp_downloaded] Error 101
make[1]: *** Waiting for unfinished jobs....
>>> ripgrep af6b6c543b224d348a8876f0c06245d9ea7929c5 Downloading
wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/output' 'https://github.com/burntsushi/ripgrep/archive/af6b6c543b224d348a8876f0c06245d9ea7929c5/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz'
[--SNIP--]
2023-01-14 14:30:16 (1.69 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/output’ saved [506803/506803]
warning: Both `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config` and `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config.toml` exist. Using `/home/ymorin/dev/buildroot/O/master/build/.ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz.X9rIVS/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.cargo/config`
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
error: failed to sync
Caused by:
failed to download packages
Caused by:
failed to download `aho-corasick v0.7.18`
Caused by:
unable to get packages from source
Caused by:
failed to parse manifest at `/home/ymorin/dev/buildroot/O/master/yem-dl/br-cargo-home/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18/Cargo.toml`
Caused by:
can't find library `aho_corasick`, rename file to `src/lib.rs` or specify lib.path
make[1]: *** [package/pkg-generic.mk:189: /home/ymorin/dev/buildroot/O/master/build/ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5/.stamp_downloaded] Error 101
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2023-01-14 13:58 ` Yann E. MORIN
@ 2023-01-14 14:17 ` Thomas Petazzoni via buildroot
2023-01-14 14:19 ` Yann E. MORIN
1 sibling, 0 replies; 10+ messages in thread
From: Thomas Petazzoni via buildroot @ 2023-01-14 14:17 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Moritz Bitsch, James Hilliard, Buildroot List
On Sat, 14 Jan 2023 14:58:17 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> Unfortunately, this causes breakage when there are parallel downloads
> (see below for except; full log as attachment).
Thanks for the report.
>
> So, there are two solutions:
>
> 1. revert to a per-package cargo home, losing the benefit of a shared
> cache, but trivial to implement
>
> 2. add locking ourselves in the post-process scripts, so that there are
> never two "cargo vendor" running in parallel, which is slightly more
> involved.
Is that really that involved?
> 3. tell upstream that they borked their locking?
We should obviously tell them.
> >>> dust 0.8.1 Downloading
> wget --passive-ftp -nd -t 3 -O '/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output' 'https://github.com/bootandy/dust/archive/v0.8.1/dust-0.8.1.tar.gz'
> [--SNIP--]
> 2023-01-14 14:30:15 (1.42 MB/s) - ‘/home/ymorin/dev/buildroot/O/master/build/.dust-0.8.1.tar.gz.ybDevZ/output’ saved [89023/89023]
>
> Blocking waiting for file lock on package cache
> Blocking waiting for file lock on package cache
So they do have locking but it doesn't work?
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Buildroot] [PATCH next 2/2] package/pkg-cargo: move CARGO_HOME into DL_DIR
2023-01-14 13:58 ` Yann E. MORIN
2023-01-14 14:17 ` Thomas Petazzoni via buildroot
@ 2023-01-14 14:19 ` Yann E. MORIN
1 sibling, 0 replies; 10+ messages in thread
From: Yann E. MORIN @ 2023-01-14 14:19 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: Moritz Bitsch, James Hilliard, Buildroot List
Thomas, All,
On 2023-01-14 14:58 +0100, Yann E. MORIN spake thusly:
> On 2022-11-03 23:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> > CARGO_HOME is where Cargo stores its downloaded artefacts. See
> > https://doc.rust-lang.org/cargo/reference/environment-variables.html:
[--SNIP--]
> > To solve this, this commit moves CARGO_HOME into $(DL_DIR), so that it
> > is shared between builds and packages.
> Unfortunately, this causes breakage when there are parallel downloads
> (see below for except; full log as attachment).
> So, there are two solutions:
> 1. revert to a per-package cargo home, losing the benefit of a shared
> cache, but trivial to implement
> 2. add locking ourselves in the post-process scripts, so that there are
> never two "cargo vendor" running in parallel, which is slightly more
> involved.
In fact, it turns out that doing the lock ourselves is relatively
trivial, so I'll do a few more tests and send a patch...
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 10+ messages in thread