* Re: [Buildroot] [PATCH] support/download: fix cargo vendoring
2023-01-09 16:30 [Buildroot] [PATCH] support/download: fix cargo vendoring Yann E. MORIN
@ 2023-01-09 21:28 ` Yann E. MORIN
2023-01-09 21:55 ` James Hilliard
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Yann E. MORIN @ 2023-01-09 21:28 UTC (permalink / raw)
To: buildroot; +Cc: Romain Naour, Simon Richter, James Hilliard
All,
On 2023-01-09 17:30 +0100, Yann E. MORIN spake thusly:
> Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0)
> forgot, despite the big comment above the version strings, to confirm
> that the vendoring was still working.
>
> Previously, we were adding the vendoring equivalence manually, but in
> commit 04154a651729 (support/download/cargo-post-process: cargo output
> for vendor config), we switched to using the output of "vargo vendor"
> (on stdout) to support caxses were the vendoring equivalence would be
> more complex (e.g. when using crates not hosted on crates.io).
>
> With rust until and including 1.65.0, "cargo vendor" would output (for
> crates.io crates) the same output as our manual fixups, except it was
> preceded by an empty line. So, to avoid recompting all our hashes, we
> added a tweak to strip away the leading empty line in 04154a651729.
>
> But rust 1.66.0 includes [0] which changes the output (on stdout) of
> "cargo vendor", where the first empty line is no longer emitted.
>
> This means that our tweak for rust 1.65.0 now strips out an important
> part of the cargo vendor output, which renders the archives invalid, and
> thus generates different archives, which fail to validate against our
> hashes.
>
> Fix this by doing what the comment in the post-process helper states,
> and just keep the whole output of "cargo vendor", byt just removing the
> "tail --lines=+2". Since that comment is no longer meaningful, we drop
> it too.
>
> Now, all our 6 cargo-based packages can be vendored again, without
> changing our hashes, but most importantly, with valid archives.
Of course, I forgot to account for the 5 python packages that have rust
code; they too are fixed with this change.
Regards,
Yann E. MORIN.
> Still, we keep the comment above the versions strings, in the hope that
> a future bumper will notice and be more careful at validating the
> vendoring.
>
> [0] https://github.com/rust-lang/cargo/pull/11273
>
> Fixes:
> http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat)
> http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust)
> http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine)
> http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep)
> ... and so many others...
>
> Reported-by: Romain Naour <romain.naour@gmail.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: James Hilliard <james.hilliard1@gmail.com>
> Cc: Simon Richter <simon.richter@ptwdosimetry.com>
> ---
> support/download/cargo-post-process | 16 +---------------
> 1 file changed, 1 insertion(+), 15 deletions(-)
>
> diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
> index 186e9eb69b..21a6be8dbe 100755
> --- a/support/download/cargo-post-process
> +++ b/support/download/cargo-post-process
> @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}"
> pushd "${base_name}" > /dev/null
>
> # Create the local .cargo/config with vendor info
> -#
> -# The first line of the output to stdout is empty.
> -# So skip it to have the file start with the vendoring
> -# configuration (`tail --lines=+2`).
> -#
> -# NOTE:
> -# There is a patch for cargo to remove the first empty line:
> -# See: https://github.com/rust-lang/cargo/pull/11273
> -#
> -# The patch already landed in +nightly and will end up
> -# in +stable soon.
> -#
> -# -> When updating rust/cargo, the call to `tail` must be removed.
> -#
> mkdir -p .cargo/
> cargo vendor \
> --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
> --locked VENDOR \
> - | tail --lines=+2 | tee .cargo/config
> + | tee .cargo/config
>
> popd > /dev/null
>
> --
> 2.25.1
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Buildroot] [PATCH] support/download: fix cargo vendoring
2023-01-09 16:30 [Buildroot] [PATCH] support/download: fix cargo vendoring Yann E. MORIN
2023-01-09 21:28 ` Yann E. MORIN
@ 2023-01-09 21:55 ` James Hilliard
2023-01-10 12:49 ` Romain Naour
2023-01-10 17:00 ` Yann E. MORIN
3 siblings, 0 replies; 6+ messages in thread
From: James Hilliard @ 2023-01-09 21:55 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Romain Naour, Simon Richter, buildroot
On Mon, Jan 9, 2023 at 9:30 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0)
> forgot, despite the big comment above the version strings, to confirm
> that the vendoring was still working.
>
> Previously, we were adding the vendoring equivalence manually, but in
> commit 04154a651729 (support/download/cargo-post-process: cargo output
> for vendor config), we switched to using the output of "vargo vendor"
> (on stdout) to support caxses were the vendoring equivalence would be
> more complex (e.g. when using crates not hosted on crates.io).
>
> With rust until and including 1.65.0, "cargo vendor" would output (for
> crates.io crates) the same output as our manual fixups, except it was
> preceded by an empty line. So, to avoid recompting all our hashes, we
> added a tweak to strip away the leading empty line in 04154a651729.
>
> But rust 1.66.0 includes [0] which changes the output (on stdout) of
> "cargo vendor", where the first empty line is no longer emitted.
>
> This means that our tweak for rust 1.65.0 now strips out an important
> part of the cargo vendor output, which renders the archives invalid, and
> thus generates different archives, which fail to validate against our
> hashes.
>
> Fix this by doing what the comment in the post-process helper states,
> and just keep the whole output of "cargo vendor", byt just removing the
> "tail --lines=+2". Since that comment is no longer meaningful, we drop
> it too.
>
> Now, all our 6 cargo-based packages can be vendored again, without
> changing our hashes, but most importantly, with valid archives.
>
> Still, we keep the comment above the versions strings, in the hope that
> a future bumper will notice and be more careful at validating the
> vendoring.
>
> [0] https://github.com/rust-lang/cargo/pull/11273
>
> Fixes:
> http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat)
> http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust)
> http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine)
> http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep)
> ... and so many others...
>
> Reported-by: Romain Naour <romain.naour@gmail.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: James Hilliard <james.hilliard1@gmail.com>
> Cc: Simon Richter <simon.richter@ptwdosimetry.com>
> ---
> support/download/cargo-post-process | 16 +---------------
> 1 file changed, 1 insertion(+), 15 deletions(-)
>
> diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
> index 186e9eb69b..21a6be8dbe 100755
> --- a/support/download/cargo-post-process
> +++ b/support/download/cargo-post-process
> @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}"
> pushd "${base_name}" > /dev/null
>
> # Create the local .cargo/config with vendor info
> -#
> -# The first line of the output to stdout is empty.
> -# So skip it to have the file start with the vendoring
> -# configuration (`tail --lines=+2`).
> -#
> -# NOTE:
> -# There is a patch for cargo to remove the first empty line:
> -# See: https://github.com/rust-lang/cargo/pull/11273
> -#
> -# The patch already landed in +nightly and will end up
> -# in +stable soon.
> -#
> -# -> When updating rust/cargo, the call to `tail` must be removed.
> -#
> mkdir -p .cargo/
> cargo vendor \
> --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
> --locked VENDOR \
> - | tail --lines=+2 | tee .cargo/config
> + | tee .cargo/config
>
> popd > /dev/null
>
> --
> 2.25.1
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Buildroot] [PATCH] support/download: fix cargo vendoring
2023-01-09 16:30 [Buildroot] [PATCH] support/download: fix cargo vendoring Yann E. MORIN
2023-01-09 21:28 ` Yann E. MORIN
2023-01-09 21:55 ` James Hilliard
@ 2023-01-10 12:49 ` Romain Naour
2023-01-10 17:02 ` Yann E. MORIN
2023-01-10 17:00 ` Yann E. MORIN
3 siblings, 1 reply; 6+ messages in thread
From: Romain Naour @ 2023-01-10 12:49 UTC (permalink / raw)
To: Yann E. MORIN, buildroot; +Cc: James Hilliard, Simon Richter, Romain Naour
Hello Yann, All,
Le 09/01/2023 à 17:30, Yann E. MORIN a écrit :
> Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0)
> forgot, despite the big comment above the version strings, to confirm
> that the vendoring was still working.
>
> Previously, we were adding the vendoring equivalence manually, but in
> commit 04154a651729 (support/download/cargo-post-process: cargo output
> for vendor config), we switched to using the output of "vargo vendor"
vargo/cargo
> (on stdout) to support caxses were the vendoring equivalence would be
caxses/cases
> more complex (e.g. when using crates not hosted on crates.io).
>
> With rust until and including 1.65.0, "cargo vendor" would output (for
> crates.io crates) the same output as our manual fixups, except it was
> preceded by an empty line. So, to avoid recompting all our hashes, we
> added a tweak to strip away the leading empty line in 04154a651729.
>
> But rust 1.66.0 includes [0] which changes the output (on stdout) of
> "cargo vendor", where the first empty line is no longer emitted.
>
> This means that our tweak for rust 1.65.0 now strips out an important
> part of the cargo vendor output, which renders the archives invalid, and
> thus generates different archives, which fail to validate against our
> hashes.
>
> Fix this by doing what the comment in the post-process helper states,
> and just keep the whole output of "cargo vendor", byt just removing the
byt/by
> "tail --lines=+2". Since that comment is no longer meaningful, we drop
> it too.
>
> Now, all our 6 cargo-based packages can be vendored again, without
> changing our hashes, but most importantly, with valid archives.
>
> Still, we keep the comment above the versions strings, in the hope that
> a future bumper will notice and be more careful at validating the
> vendoring.
>
> [0] https://github.com/rust-lang/cargo/pull/11273
We have some runtime test for gitlab/github download wrapper, do we want a test
for cargo vendor too?
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Best regards,
Romain
>
> Fixes:
> http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat)
> http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust)
> http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine)
> http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep)
> ... and so many others...
>
> Reported-by: Romain Naour <romain.naour@gmail.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: James Hilliard <james.hilliard1@gmail.com>
> Cc: Simon Richter <simon.richter@ptwdosimetry.com>
> ---
> support/download/cargo-post-process | 16 +---------------
> 1 file changed, 1 insertion(+), 15 deletions(-)
>
> diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
> index 186e9eb69b..21a6be8dbe 100755
> --- a/support/download/cargo-post-process
> +++ b/support/download/cargo-post-process
> @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}"
> pushd "${base_name}" > /dev/null
>
> # Create the local .cargo/config with vendor info
> -#
> -# The first line of the output to stdout is empty.
> -# So skip it to have the file start with the vendoring
> -# configuration (`tail --lines=+2`).
> -#
> -# NOTE:
> -# There is a patch for cargo to remove the first empty line:
> -# See: https://github.com/rust-lang/cargo/pull/11273
> -#
> -# The patch already landed in +nightly and will end up
> -# in +stable soon.
> -#
> -# -> When updating rust/cargo, the call to `tail` must be removed.
> -#
> mkdir -p .cargo/
> cargo vendor \
> --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
> --locked VENDOR \
> - | tail --lines=+2 | tee .cargo/config
> + | tee .cargo/config
>
> popd > /dev/null
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Buildroot] [PATCH] support/download: fix cargo vendoring
2023-01-10 12:49 ` Romain Naour
@ 2023-01-10 17:02 ` Yann E. MORIN
0 siblings, 0 replies; 6+ messages in thread
From: Yann E. MORIN @ 2023-01-10 17:02 UTC (permalink / raw)
To: Romain Naour; +Cc: James Hilliard, Simon Richter, Romain Naour, buildroot
Romain, All,
On 2023-01-10 13:49 +0100, Romain Naour spake thusly:
> Le 09/01/2023 à 17:30, Yann E. MORIN a écrit :
[--SNIP--]
> > Still, we keep the comment above the versions strings, in the hope that
> > a future bumper will notice and be more careful at validating the
> > vendoring.
> We have some runtime test for gitlab/github download wrapper, do we want a test
> for cargo vendor too?
Yes, that would be nice, indeed.
> Reviewed-by: Romain Naour <romain.naour@smile.fr>
> Tested-by: Romain Naour <romain.naour@smile.fr>
Typoes fixed when applying, thanks! ;-)
Regards,
Yann E. MORIN.
> Best regards,
> Romain
>
> >
> > Fixes:
> > http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat)
> > http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust)
> > http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine)
> > http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep)
> > ... and so many others...
> >
> > Reported-by: Romain Naour <romain.naour@gmail.com>
> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > Cc: James Hilliard <james.hilliard1@gmail.com>
> > Cc: Simon Richter <simon.richter@ptwdosimetry.com>
> > ---
> > support/download/cargo-post-process | 16 +---------------
> > 1 file changed, 1 insertion(+), 15 deletions(-)
> >
> > diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
> > index 186e9eb69b..21a6be8dbe 100755
> > --- a/support/download/cargo-post-process
> > +++ b/support/download/cargo-post-process
> > @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}"
> > pushd "${base_name}" > /dev/null
> >
> > # Create the local .cargo/config with vendor info
> > -#
> > -# The first line of the output to stdout is empty.
> > -# So skip it to have the file start with the vendoring
> > -# configuration (`tail --lines=+2`).
> > -#
> > -# NOTE:
> > -# There is a patch for cargo to remove the first empty line:
> > -# See: https://github.com/rust-lang/cargo/pull/11273
> > -#
> > -# The patch already landed in +nightly and will end up
> > -# in +stable soon.
> > -#
> > -# -> When updating rust/cargo, the call to `tail` must be removed.
> > -#
> > mkdir -p .cargo/
> > cargo vendor \
> > --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
> > --locked VENDOR \
> > - | tail --lines=+2 | tee .cargo/config
> > + | tee .cargo/config
> >
> > popd > /dev/null
> >
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] support/download: fix cargo vendoring
2023-01-09 16:30 [Buildroot] [PATCH] support/download: fix cargo vendoring Yann E. MORIN
` (2 preceding siblings ...)
2023-01-10 12:49 ` Romain Naour
@ 2023-01-10 17:00 ` Yann E. MORIN
3 siblings, 0 replies; 6+ messages in thread
From: Yann E. MORIN @ 2023-01-10 17:00 UTC (permalink / raw)
To: buildroot; +Cc: Romain Naour, Simon Richter, James Hilliard
All,
On 2023-01-09 17:30 +0100, Yann E. MORIN spake thusly:
> Commit de4cf253752d (package/{rust, rust-bin}: bump to version 1.66.0)
> forgot, despite the big comment above the version strings, to confirm
> that the vendoring was still working.
>
> Previously, we were adding the vendoring equivalence manually, but in
> commit 04154a651729 (support/download/cargo-post-process: cargo output
> for vendor config), we switched to using the output of "vargo vendor"
> (on stdout) to support caxses were the vendoring equivalence would be
> more complex (e.g. when using crates not hosted on crates.io).
>
> With rust until and including 1.65.0, "cargo vendor" would output (for
> crates.io crates) the same output as our manual fixups, except it was
> preceded by an empty line. So, to avoid recompting all our hashes, we
> added a tweak to strip away the leading empty line in 04154a651729.
>
> But rust 1.66.0 includes [0] which changes the output (on stdout) of
> "cargo vendor", where the first empty line is no longer emitted.
>
> This means that our tweak for rust 1.65.0 now strips out an important
> part of the cargo vendor output, which renders the archives invalid, and
> thus generates different archives, which fail to validate against our
> hashes.
>
> Fix this by doing what the comment in the post-process helper states,
> and just keep the whole output of "cargo vendor", byt just removing the
> "tail --lines=+2". Since that comment is no longer meaningful, we drop
> it too.
>
> Now, all our 6 cargo-based packages can be vendored again, without
> changing our hashes, but most importantly, with valid archives.
>
> Still, we keep the comment above the versions strings, in the hope that
> a future bumper will notice and be more careful at validating the
> vendoring.
>
> [0] https://github.com/rust-lang/cargo/pull/11273
>
> Fixes:
> http://autobuild.buildroot.org/results/bea/beac7674bbc9fd2f8777b5861f65afee9c485753/ (bat)
> http://autobuild.buildroot.org/results/d1e/d1ec1ebbde115628a4b8b9099544347242a97c1c/ (dust)
> http://autobuild.buildroot.org/results/f96/f968be895be9ca98b314fdd688ef8d3bdf4e5dfb/ (hyerfine)
> http://autobuild.buildroot.org/results/a0c/a0cdb6cc9493f5248d98f98b13da854e12adc2be/ (ripgrep)
> ... and so many others...
>
> Reported-by: Romain Naour <romain.naour@gmail.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: James Hilliard <james.hilliard1@gmail.com>
> Cc: Simon Richter <simon.richter@ptwdosimetry.com>
Applied to master, with all the typoes noticed by Romain fixed, thanks.
Regards,
Yann E. MORIN.
> ---
> support/download/cargo-post-process | 16 +---------------
> 1 file changed, 1 insertion(+), 15 deletions(-)
>
> diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
> index 186e9eb69b..21a6be8dbe 100755
> --- a/support/download/cargo-post-process
> +++ b/support/download/cargo-post-process
> @@ -25,25 +25,11 @@ post_process_unpack "${base_name}" "${output}"
> pushd "${base_name}" > /dev/null
>
> # Create the local .cargo/config with vendor info
> -#
> -# The first line of the output to stdout is empty.
> -# So skip it to have the file start with the vendoring
> -# configuration (`tail --lines=+2`).
> -#
> -# NOTE:
> -# There is a patch for cargo to remove the first empty line:
> -# See: https://github.com/rust-lang/cargo/pull/11273
> -#
> -# The patch already landed in +nightly and will end up
> -# in +stable soon.
> -#
> -# -> When updating rust/cargo, the call to `tail` must be removed.
> -#
> mkdir -p .cargo/
> cargo vendor \
> --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
> --locked VENDOR \
> - | tail --lines=+2 | tee .cargo/config
> + | tee .cargo/config
>
> popd > /dev/null
>
> --
> 2.25.1
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread