Re: [OE-core] [PATCH V4 2/3] rust: Upgrade 1.82.0->1.83.0 --> bloat alert !!

[Randy MacLeod <randy.macleod@windriver.com>]

[-- Attachment #1: Type: text/plain, Size: 6848 bytes --]

On 2025-03-14 11:50 a.m., Yash Shinde wrote:
>
>
> On 14-03-2025 19:18, Randy MacLeod wrote:

...


>> Thanks Yash!
>>
>> It's a bit odd that you see an 8G diff and YP AB saw 5 GB but that's 
>> close enough for now.
>>
>> Please run the same analysis for 1.84 update both before and after 
>> removing the src/gcc .
>>
>> Please summarize the data for 1.82, 1.83 and the two sets of data for 
>> 1.84.
>>
>> I'd expect 1.84 + "rm -rf src/gcc" to be somewhat larger than 1.82 
>> but much smaller than 1.83.
>>
> TMDIR FS usage                v1.82.0                          v1.83.0 
>                    v1.84.0                          v1.84 + rm -rf src/gcc
> du -sh poky/build/tmp/           65 G                               73 
> G                 73 G 72 G
>
What steps did you take (bitbake rust-native ? ) before comparing these 
tmp dir trees?

So what made the 1.82.0 and either 1.83.0 or 1.84.0 differ in size so 
significantly? See below for my approach for the src tree.
Hopefully you can do something similar or better for the build/tmp dir 
that's a ~8 GB diff from 1.82 to 1.84.
And still 7 GB after rm -rf src/gcc. Please read the rest of this email 
and send the update if it's ready before
spending time on the analysis of the size difference.

> du -sh  poky/build/../1.8x.0/rustc-1.8x.0-src/ 9.8 
> G                             12 G                                12 
> G                                           11 G
>
> The size of TMPDIR increased from 65 G in 1.82 to 73 G in 1.83 and 
> remained same in 1.84.
> After removing "src/gcc"(1.1 G) the size is reduced to 72G.
>
And the build and test worked well, I assume.

If so please send the update with the rm -rf src/gcc .



I took a look at the source tree differences a little.
I tried using ncdu and meld but they aren't really the right tool.

I ended up doing the following:

Unpack two tarballs then mkdir du

❯  du -sk rustc-1.82.0-src/* | sort -n > du/du-1.82-top

❯  du -sk rustc-1.84.0-src/* | sort -n > du/du-1.84-top

# manually edit line order


*❯* sdiff du/du-1.8*-top
4	rustc-1.82.0-src/Cargo.toml			      |	4	rustc-1.84.0-src/Cargo.toml
4	rustc-1.82.0-src/configure			      |	4	rustc-1.84.0-src/configure
4	rustc-1.82.0-src/CONTRIBUTING.md		      |	4	rustc-1.84.0-src/CONTRIBUTING.md
4	rustc-1.82.0-src/git-commit-hash		      |	4	rustc-1.84.0-src/git-commit-hash
4	rustc-1.82.0-src/git-commit-info		      |	4	rustc-1.84.0-src/git-commit-info
4	rustc-1.82.0-src/LICENSE-MIT			      |	4	rustc-1.84.0-src/LICENSE-MIT
4	rustc-1.82.0-src/README.md			      |	4	rustc-1.84.0-src/README.md
4	rustc-1.82.0-src/version			      |	4	rustc-1.84.0-src/version
4	rustc-1.82.0-src/x.py				      |	4	rustc-1.84.0-src/x.py
12	rustc-1.82.0-src/LICENSE-APACHE			      |	12	rustc-1.84.0-src/LICENSE-APACHE
24	rustc-1.82.0-src/COPYRIGHT			      |	24	rustc-1.84.0-src/COPYRIGHT
40	rustc-1.82.0-src/config.example.toml		      |	44	rustc-1.84.0-src/config.example.toml
140	rustc-1.82.0-src/Cargo.lock			      |	140	rustc-1.84.0-src/Cargo.lock
736	rustc-1.82.0-src/RELEASES.md			      |	768	rustc-1.84.0-src/RELEASES.md
36364	rustc-1.82.0-src/compiler			      |	37124	rustc-1.84.0-src/compiler
43928	rustc-1.82.0-src/library			      |	44484	rustc-1.84.0-src/library
165588	rustc-1.82.0-src/tests				      |	169872	rustc-1.84.0-src/tests
1379288	rustc-1.82.0-src/src				      |	2889512	rustc-1.84.0-src/src
1975056	rustc-1.82.0-src/vendor				      |	1926968	rustc-1.84.0-src/vendor

Since I'm always asking for you to do the math, I got chatgtp to do mine 
for me:

|+4 config.example.toml +32 RELEASES.md +760 compiler +556 library +4284 
tests +1510224 src -48088 vendor|

Positive numbers indicate an increase in size, while the negative number 
for |vendor| indicates a decrease.


# Similarly for src and vendor subdirs:

❯  du -sk rustc-1.84.0-src/src/* | sort -n > du/du-1.84-src
...


sdiff du/du-1.8*-src
4	rustc-1.82.0-src/src/README.md			      |	4	rustc-1.84.0-src/src/README.md
4	rustc-1.82.0-src/src/version			      |	4	rustc-1.84.0-src/src/version
44	rustc-1.82.0-src/src/stage0			      |	44	rustc-1.84.0-src/src/stage0
							      |	56	rustc-1.84.0-src/src/build_helper
							      |	56	rustc-1.84.0-src/src/rustc-std-workspace
60	rustc-1.82.0-src/src/rustdoc-json-types		      |	64	rustc-1.84.0-src/src/rustdoc-json-types
1208	rustc-1.82.0-src/src/ci				      |	1220	rustc-1.84.0-src/src/ci
1408	rustc-1.82.0-src/src/bootstrap			      |	1468	rustc-1.84.0-src/src/bootstrap
2348	rustc-1.82.0-src/src/etc			      |	2344	rustc-1.84.0-src/src/etc
3368	rustc-1.82.0-src/src/librustdoc			      |	3456	rustc-1.84.0-src/src/librustdoc
42096	rustc-1.82.0-src/src/doc			      |	43048	rustc-1.84.0-src/src/doc
290368	rustc-1.82.0-src/src/tools			      |	693384	rustc-1.84.0-src/src/tools
1038376	rustc-1.82.0-src/src/llvm-project		      |	1038408	rustc-1.84.0-src/src/llvm-project
							      >	1105952	rustc-1.84.0-src/src/gcc


the major difference is the src dir.

|+56 build_helper +56 rustc-std-workspace +4 rustdoc-json-types +12 ci 
+60 bootstrap -4 etc +88 librustdoc +952 doc +403016 tools +32 
llvm-project +1105952 gcc |

Vendor dir is harder to compare since there are so many files and the number of files has changed:

✦ ❯ wc -l du/du-1.82-vendor du/du-1.84-vendor
   1511 du/du-1.82-vendor
   1530 du/du-1.84-vendor


The total size changed from 1975056 to 1926968 - a decrease, so that's not worth looking into.

../Randy



> Thanks,
> Yash.
>
>>
>> ../Randy
>>
>>
>>>
>>> The main reason for FS usuage differences in v1.83 are:
>>> 1. Addition of gcc  ~1.1 G.2. The addition of enzyme AD sub module ~392 Mb.
>>>
>>> References:
>>> https://github.com/rust-lang/rust/commit/9028177115718b38c0a189c4fb9eda03a14841d7
>>> https://github.com/rust-lang/rust/issues/124509
>>> https://github.com/rust-lang/rust/pull/129175 
>>> https://github.com/rust-lang/rust/commit/59d4114b2d1aaac9a6dfe770997f2e79ccfd28ab
>>>
>>>
>>> Dir
>>> 	1.82.0
>>> 	1.83.0
>>> 	Diff
>>> rustc-1.8x.0-src.tar.xz
>>> 	210Mb
>>> 	338Mb
>>> 	128Mb
>>> rustc-1.8x.0-src
>>> 	3.5G
>>> 	4.8G
>>> 	1.3G
>>>
>>> (PS: Please ignore the previous mail.)
>>>
>>> Regards,
>>> Yash
>>>>
>>>>
>>>> ../Randy
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Richard
>>>>>
>>>>>
>>>>
>>>> -- 
>>>> # Randy MacLeod
>>>> # Wind River Linux
>>
>>
>> -- 
>> # Randy MacLeod
>> # Wind River Linux
> -- 
> Regards,
> Yash Shinde


-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 14340 bytes --]