Re: [OE-core] GCC crashes on aarch64 since gatesgarth

["Max Krummenacher" <max.oss.09@gmail.com>]

Hi Mike

Am Donnerstag, den 11.03.2021, 10:43 +0100 schrieb Mike Looijmans:
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> System Expert
> 
> 
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
> 
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> On 05-03-2021 00:32, Randy MacLeod wrote:
> > On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
> > > I've been doing quite some tests and things last week, my conclusion so far 
> > > is that the latest gcc seems to trigger a bug particular to this CPU. Maybe 
> > > even this particular batch.
> > > 
> > > Hours of testing found no issues with RAM.
> > > 
> > > I've also done huge amounts of builds with zeus, no problems.
> > > 
> > > Then googling for "illegal instruction gcc ryzen 1700" reveals quite a few 
> > > threads on something that sounds suspiciously similar. Though it makes me 
> > > wonder why I never encountered it before, since these are all many years 
> > > old. I did try a bunch of the suggestions done there, but so far no luck.
> > > 
> > > 
> > 
> > Any update on this bug Mike?
> > Have you opened a YP bug? If not please do.
> > I'd like to see if the problem still happens on master.
> 
> So far I've discovered that this only happens when compiling for an aarch64. 
> When targetting 32-bit ARM the compiler crash does not occur.
> 
> I've contacted AMD for an RMA for my CPU, but it's out of warranty now. I plan 
> to borrow a compatible CPU to see if that makes the problem go away. If it 
> does I'll just buy a new CPU...
> 
> As an example, if I take the current gatesgarth branch and meta-raspberrypi, 
> then building for "raspberrypi4" runs without a hitch. If I chose 
> "raspberrypi4-64" as MACHINE, the build will crash multiple times, typically 
> on large components like the linux kernel, qt and boost. It happens on both 
> plain C (kernel, u-boot) and on C++ code (boost, Qt). The same applies to e.g. 
> the Xilinx Zynq, building for the 7-series (cortex A9) is fine but building 
> for the MPSoC (cortex A53) will crash.
> 
> A silly workaround is to run "while ! bitbake ... ; do echo again; done" which 
> will retry until it succeeds.
> 
> I'd be very interested if there's anyone with Ryzen setup who can reproduce 
> this, or fail to, which is also interesting to know.
> 
> I also tried building in tmpfs and on a different SSD to rule out the NVME as 
> a cause.
> 
> I'll try the master branch too, and report back on that (tomorrow probably).
> 
> 
> > ../Randy
> > 
> > 
> > > Met vriendelijke groet / kind regards,
> > > 
> > > Mike Looijmans
> > > System Expert
> > > 
> > > 
> > > TOPIC Embedded Products B.V.
> > > Materiaalweg 4, 5681 RJ Best
> > > The Netherlands
> > > 
> > > T: +31 (0) 499 33 69 69
> > > E: mike.looijmans@topicproducts.com
> > > W: www.topicproducts.com
> > > 
> > > Please consider the environment before printing this e-mail
> > > On 27-01-2021 16:10, Mike Looijmans via lists.openembedded.org wrote:
> > > > When doing large builds, the GCC compiler tends to crash on random spots in 
> > > > the code. There are a few common denominators though.
> > > > 
> > > > It only happens when compiling for aarch64 (cortex-A53), not for 32-bit arm 
> > > > (cortex-A9)
> > > > 
> > > > It's random and usually happens on "big" sets like kernel, openssl, boost, 
> > > > u-boot etc.
> > > > 
> > > > It always reports "during GIMPLE pass: ealias" in the error, for example:
> > > > 
> > > > > during GIMPLE pass: ealias
> > > > > ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
> > > > > ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler error: 
> > > > Illegal instruction
> > > > > 1239 | }
> > > > 
> > > > Compiling the same thing again usually goes fine.
> > > > 
> > > > I've never experienced this with the zeus and older branches of OE.
> > > > 
> > > > 
> > > > I've already tried upgrading to the latest gatesgarth status, and cleaning 
> > > > out everything and start from scratch. I've also run "mprime" test on my 
> > > > machine (over one hour) just to be confident that the system itself is 
> > > > really okay.
> > > > 
> > > > 
> > > > Ideas to diagnose, fix or reliably reprodruce are more than welcome. 
> > > 
> > > 

For me a build of core-image-weston on a AMD Ryzen 7 3700X succeeded. The build run trough without
any hick-ups. openssl-1.1.1i is built as part of that image.
I used HEAD of today's gatesgarth.
So the issue seems to not affect all Ryzen CPUs.

Max


MACHINE=raspberrypi4-64 bitbake core-image-weston
WARNING: Host distribution "opensuseleap-15.2" has not been validated with this version of the build
system; you may possibly experience unexpected failures. It is recommended that you use a tested
distribution.
Loading cache: 100%
|                                                                                                   
                                                                            | ETA:  --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |#####################################################| Time: 0:00:43
Parsing of 2884 .bb files complete (0 cached, 2884 parsed). 4160 targets, 288 skipped, 0 masked, 0
errors.
WARNING: No recipes in default available for:
  /mnt/devel/oe-disk/oe-core_gatesgarth/build/../layers/meta-raspberrypi/recipes-
multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.%.bbappend
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-tdx-linux"
MACHINE              = "raspberrypi4-64"
DISTRO               = "tdx-xwayland"
DISTRO_VERSION       = "0.0.0-devel-20210311125851+build.0"
TUNE_FEATURES        = "aarch64 armv8a crc crypto cortexa72"
TARGET_FPU           = ""
meta-toradex-nxp     = "HEAD:b8187ed55f204f50dea60c27f6cc79e9d2f1081b"
meta-freescale       = "gatesgarth:1acf098972f9f86b26eee815827667dbd2fcdeaf"
meta-freescale-3rdparty = "gatesgarth:efef93a0dc5be90c5bdc81dae252516887c95586"
meta-toradex-tegra   = "HEAD:627949f52b58d54ec2b5992b327bb33123cc53b2"
meta-toradex-bsp-common = "HEAD:e06c552ef433689b12caef0bf1f589d0cab180f9"
meta-raspberrypi     = "master:fdf02accfe3f68d11fa178d72eb377f9ca96411c"
meta-oe              
meta-filesystems     
meta-gnome           
meta-xfce            
meta-initramfs       
meta-networking      
meta-multimedia      
meta-python          = "gatesgarth:945f062ff098dc9c8ba8d22c5eef88adec60730d"
meta-freescale-distro = "gatesgarth:50eb2b32e7702bc435049bfe0a98fc65c864c106"
meta-toradex-demos   = "HEAD:4ffac94ed18c8701511385dcfc363a3051855fd7"
meta-qt5             = "gatesgarth:2b33a5d5e888370bb56685b86aa82b73624f19f0"
meta-toradex-distro  = "HEAD:6ddae219515fda5e6ffb0ad6a2fdc5430d9fa926"
meta-poky            = "gatesgarth:ac4a956f606d9cb4e025af80e6f4dcb74cbfd941"
meta                 = "gatesgarth:6311cb4930bd0add7aec61e5e0df6bb7ae0c4481"

Initialising tasks: 100%
|###########################################################################| Time: 0:00:04
Sstate summary: Wanted 1109 Found 142 Missed 967 Current 702 (12% match, 46% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 5358 tasks of which 2483 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 12 seconds

Summary: There were 2 WARNING messages shown.