[TEST] amt.sh crashes smcrouted

[TEST] amt.sh crashes smcrouted

[Jakub Kicinski]

Hi Taehee Yoo!

We started running amt tests in the netdev CI, and it looks like it
hangs - or at least it doesn't produce any output for long enough
for the test runner to think it hung.

While looking at the logs, however, I see:

[    3.361660] smcrouted[294]: segfault at 7fff480c95f3 ip 00000000004034e4 sp 00007fff480b9410 error 6 in smcrouted[402000+a000] likely on CPU 3 (core 3, socket 0)
[    3.361812] Code: 74 24 38 89 ef e8 4c 33 00 00 44 0f b7 f8 66 39 84 24 e2 01 00 00 75 09 45 85 ed 0f 85 ed 01 00 00 48 8b 44 24 38 0f b6 40 33 <42> 88 84 3c e4 01 00 00 48 8b 3b 48 8d 54 24 38 48 8d 74 24 50 e8

https://netdev-3.bots.linux.dev/vmksft-net/results/577882/4-amt-sh/

So I think the cause may be a bug in smcroute.

We use smcroute build from latest git
# cd smcroute/
# git log -1 --oneline
cd25930 .github: use same CFLAGS for both configure runs
# smcroute -v
SMCRoute v2.5.6

Could you check if you can repro this crash?

Re: [TEST] amt.sh crashes smcrouted

[Taehee Yoo]

On Fri, May 3, 2024 at 5:00 AM Jakub Kicinski <kuba@kernel.org> wrote:
>

Hi Jakub,
Thanks a lot for the report!

> Hi Taehee Yoo!
>
> We started running amt tests in the netdev CI, and it looks like it
> hangs - or at least it doesn't produce any output for long enough
> for the test runner to think it hung.
>
> While looking at the logs, however, I see:
>
> [    3.361660] smcrouted[294]: segfault at 7fff480c95f3 ip 00000000004034e4 sp 00007fff480b9410 error 6 in smcrouted[402000+a000] likely on CPU 3 (core 3, socket 0)
> [    3.361812] Code: 74 24 38 89 ef e8 4c 33 00 00 44 0f b7 f8 66 39 84 24 e2 01 00 00 75 09 45 85 ed 0f 85 ed 01 00 00 48 8b 44 24 38 0f b6 40 33 <42> 88 84 3c e4 01 00 00 48 8b 3b 48 8d 54 24 38 48 8d 74 24 50 e8
>
> https://netdev-3.bots.linux.dev/vmksft-net/results/577882/4-amt-sh/
>
> So I think the cause may be a bug in smcroute.
>
> We use smcroute build from latest git
> # cd smcroute/
> # git log -1 --oneline
> cd25930 .github: use same CFLAGS for both configure runs
> # smcroute -v
> SMCRoute v2.5.6
>
> Could you check if you can repro this crash?

I tried to reproduce the latest version of the smcrouted crash,
but I couldn't reproduce it.
I'm sure this crash is the reason for the failure of your case.
But the real bug of this phenomenon is the amt.sh doesn't have timeout logic.
If the smcrouted did crash or it couldn't finish in time something in it,
it should print FAIL and then quit this test, but it waits forever.

I will send a patch for it.

Thank you so much for taking care of it.
Taehee Yoo