Re: [OE-core] [PATCH V4 2/3] rust: Upgrade 1.82.0->1.83.0 --> bloat alert !!

[Randy MacLeod <randy.macleod@windriver.com>]

[-- Attachment #1: Type: text/plain, Size: 4605 bytes --]

On 2025-03-05 8:44 a.m., Varatharajan, Deepesh via 
lists.openembedded.org wrote:
> From: Deepesh Varatharajan<Deepesh.Varatharajan@windriver.com>
>
> Rust stable version updated to 1.83.0.
> https://blog.rust-lang.org/2024/11/28/Rust-1.83.0.html
>
> Renamed and modified the below patch to adapt the new version.
> rv32-cargo-rustix-0.38.34-fix.patch->rv32-cargo-rustix-0.38.37-fix.patch
>
> Modified the below patches to adapt the new version.
> repro-issue-fix-with-cc-crate-hashmap.patch
> revert-link-std-statically-in-rustc_driver-feature.patch
>
> Dropped: zlib-off64_t.patch
> https://github.com/madler/zlib/commit/a566e156b3fa07b566ddbf6801b517a9dba04fa3kq
>
> Because of the following commit ,
> https://github.com/rust-lang/rust/commit/68034f837a39387e49fc7d7c5b088f5372a1127e
> when we enable lib32, getting build failure because there is a check for target
> support for "-Zdual-proc-macros" flag not functioning properly when lib32 is
> enabled in the build environment. So for now reverting this commit and bring
> back the previous behavior, where the "-Zdual-proc-macros" flag is always
> added for building proc macros, regardless of the target architecture's support.
> This would bypass the check introduced in the patch, allowing the build to
> proceed without error, even when building for a 64-bit architecture with lib32 enabled.


Summary:

rust-1.83 (and 1.84)  has a full gcc-14 tree, bloating the tar.xz by 128 
Mb, unpacked sources by 1.3 GB.
The build impact seems to be 5 GB . This was an upstream error that has 
been fixed in 1.85.

If you think this is unacceptable, please read on and reply to my 
suggestions about what we could do.


Details:

Richard pointed out that the buildperf info here:
https://valkyrie.yocto.io/pub/non-release/20250307-81/testresults/buildperf-alma8/perf-alma8-vk_master_20250307150029_5323603048.html 


shows that rust 1.83 added 5GB to TMPDIR disk usage !!

The Rust .tar.xz went from 210M -> 337 MB, 1.84 and I checked 1.85 is 
down a bit to 261 MB.

❯ ls -sh *xz| cat
210M rustc-1.82.0-src.tar.xz
338M rustc-1.83.0-src.tar.xz
339M rustc-1.84.0-src.tar.xz
262M rustc-1.85.0-src.tar.xz

❯ for i in `ls -d *src`; do du -sh $i; done
3.5G    rustc-1.82.0-src
4.8G    rustc-1.83.0-src
4.9G    rustc-1.84.0-src
3.9G    rustc-1.85.0-src

I didn't check for the other 4 GB yet...


Upstream added src/gcc in 1.83. Yikes!
❯ cat rustc-1.83.0-src/src/gcc/gcc/BASE-VER
14.0.1

A full compiler tree - not fun! :-(

In rust 1.85  src/gcc was removed.

Here's the gcc removal commit:
https://github.com/rust-lang/rust/pull/135658

The "(yet)" part of this is a bit worrying:  "... we should not need it 
for building anything in the tarballs (yet)"


Given Rust's policy of "just update to the latest and only supported 
release", they're not likely going to re-spin the 1.83/1.84 tarballs
but we should try to get a 1.83.1 and 1.84.1 anyway.

If upstream doesn't fix re-spin:

Should we do that an host the tarball  on yoctoproject.org?
Should we write a custom unpack rule to --exclude=rustc-1.83.0-src/src/gcc ?


We need to understand why they think they might eventually need to 
bundle gcc (bootstrapping?) and explain:
   1. the yocto/OE use case and how painful it is for developers who 
update daily, and pay a huge rust tax, in time, disk space, and 
electricity use
        not to mention the download tax it places on the YP infrastructure.

       Maybe if they know where we are coming from, they can show a bit 
of compassion and understanding.

   2. that they should be making smaller tarballs for general users
   3. that they should be working on up-streaming their llvm diffs so 
that the tarball can actually shrink.

I'll work on that next week.

Note that 1.85 is ~300 MB bigger thanks to a new tool:
https://enzyme.mit.edu/

which  takes arbitrary existing code as LLVM IR and computes the 
derivative (and gradient) of that function.
I don't know what that really means but apparently it's used for code 
minimization and therefore run-time optimization.
Most of that 300 MB is a benchmark... sigh.

Deepesh,

Please check for this sort of significant size changes and mention it to 
people when you notice it.
Also please open an issue for enzyme to ask if the benchmark data can be 
kept in a separate crate.

Next week, I'll also dig into the 'tmp' usage increase to understand 
where the 5 GB that Richard reported comes from.


All for today,

../Randy



[-- Attachment #2: Type: text/html, Size: 6908 bytes --]