From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07BBEC7EE2C for ; Thu, 18 May 2023 15:02:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231132AbjERPCz (ORCPT ); Thu, 18 May 2023 11:02:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230356AbjERPCw (ORCPT ); Thu, 18 May 2023 11:02:52 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C540121 for ; Thu, 18 May 2023 08:02:24 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-560ee0df572so33639387b3.0 for ; Thu, 18 May 2023 08:02:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684422073; x=1687014073; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=COEcCFWgCbplE1MtlqGXZIriKuoSjyaM69XcEc43hFs=; b=V4t31ISM6ddijUWJrMbzNdsub50JqaWK0bG2NFCCpmTFivM5lhHY9OWHzMC0C/HCMT Ja37GZLQK+L91ygKXO4cJFqPAoIr3jFi1MTJXKDiS/EGFspP9Frt+o9dDL8Nt0rXY7Ua Ausy7uWZTyz5+fN5nyBn1wmlxoKRTBGWLWJlig0yVzR79vIUv/RMFOFcXJiDrmQZoa6d HQy+bLYpiwmqRs3B2j7jXF2JXzKsYXuDfjdrnYJNbxh/Km+HRilenhnoRg1cmJFynjeP xxeuMJ42jYGmeu3M3IZ8qsQ+6U1GzhFBwKRh08c9gxKDFBu4fbkUTLHLge/Lo83UhALz gvEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684422073; x=1687014073; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=COEcCFWgCbplE1MtlqGXZIriKuoSjyaM69XcEc43hFs=; b=SdIFn4N+tvM++Z+R+4+64mHx64Cqcr9w2mwAX7Kvh26I7q0FPJBnfBQpyZaqLg1C6Z j2locWs4IW0cSqIrsDesdUOzQ8BbzOyNwHLAOIyxnsS/RQ9a/PaSq24Qv5KzNIoF1S/v 6C9xSiprNIFZHJm5YtS5aNlNf8mwJUIdi5hroaSHN3YUj44WMlrMTseFywlEWcwg3Uts 3Q2EHk+ELdhOU+Mvtqc6TsCE+K9CtsEzwKwcE5ZcUOPhjYxHng3kEvszzJOIaYgah00t rIgDgMQ7Ec8DKZnq+n6d4sqCLu481XsmByMgsTmO0QuBMWOnoY1WALWoonEoe1WJV8Ey btUA== X-Gm-Message-State: AC+VfDzwUSYI30BbYG21cNbXqoBZtkGIdDoqdL543/IrCGj6CgQVJ3Re 0hhzM8kppQNVXSH0zAasFdjPyyWazTw= X-Google-Smtp-Source: ACHHUZ7IX6Wx0LSBkpzQqjr+OXaJS2pKyVuTTzwb3z0a97s/un34w9qdMEO5QA63WIOBuZvYpFhahx6OOmI= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a81:e50e:0:b0:561:3149:d684 with SMTP id s14-20020a81e50e000000b005613149d684mr1180910ywl.10.1684422072950; Thu, 18 May 2023 08:01:12 -0700 (PDT) Date: Thu, 18 May 2023 08:01:11 -0700 In-Reply-To: Mime-Version: 1.0 References: <71288f04-546c-9af3-e29a-eb3c44e0d948@gmail.com> Message-ID: Subject: Re: Fwd: Persistent rt_sigreturn segfaults on KVM VMs after upgrade to 5.15 From: Sean Christopherson To: Bagas Sanjaya Cc: Linux Kernel Mailing List , Linux Regressions , Linux KVM , Paolo Bonzini , Theodor Milkov Content-Type: text/plain; charset="us-ascii" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, May 18, 2023, Bagas Sanjaya wrote: > On 5/18/23 20:57, Bagas Sanjaya wrote: > > Hi, > > > > I notice a regression report on Bugzilla [1]. Quoting from it: > > > >> I'm experiencing sporadic but persistent segmentation faults on the KVM > >> VMs I manage. These faults began appearing after upgrading from Linux > >> Kernel 4.x to 5.15.59. I further upgraded to 5.15.91 and transitioned the > >> userspace from Debian 10 (buster) to Debian 11 (bullseye), yet the issues > >> persist. Notably, the libc has also changed in the process as seen in the > >> following error logs: Was the host or guest kernel upgraded? If the guest kernel was upgraded, it's unlikely, though still possible, that this is a KVM bug. > >> post.sh[21952]: bad frame in rt_sigreturn frame:000072db65961bb8 > >> ip:6c25f82a9a5d sp:72db65962168 orax:ffffffffffffffff in > >> libc-2.28.so[6c25f8294000+147000] > >> > >> cron[7626]: bad frame in rt_sigreturn frame:000073ddebeb6ff8 > >> ip:72ad9f44d594 sp:73ddebeb75a8 orax:ffffffffffffffff in > >> libc-2.28.so[72ad9f3a9000+147000] > >> > >> cron[64687]: bad frame in rt_sigreturn frame:000073265764b038 > >> ip:67c7b5a0f14a sp:73265764b5f0 orax:ffffffffffffffff in > >> libc-2.31.so[67c7b596f000+159000] > >> > >> worker.py[54568]: bad frame in rt_sigreturn frame:000078eef6591cf8 > >> ip:6c9f9b2a604e sp:78eef6592298 orax:ffffffffffffffff in > >> libpthread-2.31.so[6c9f9b29a000+10000] > >> > >> > >> The segmentation faults occur 1-3 times daily across approximately 1000 > >> VMs running on hundreds of (supermicro, intel cpu) bare-metal servers. > >> Currently, there's no reliable way for me to reproduce the issue. I > >> initially considered this bug - > >> https://www.spinics.net/lists/linux-tip-commits/msg61293.html - as a > >> possible cause, but judging from the comments it likely isn't. > >> > >> The best approximation to a reproducer I have is a Python script that > >> initiates several child processes and continuously sends them a sigusr1 > >> signal. Still, it takes a few hours to trigger the issue even when running > >> this script on several hundred VMs. > >> > >> Switching to the 6.x kernel isn't immediately feasible as these are > >> production systems with specific requirements. The transition is planned > >> but will likely take several months. > >> > >> I'm looking for suggestions on how to more reliably reproduce this > >> problem. Then I could try different old and new kernels and maybe narrow > >> it down. > > > > See bugzilla for the full thread. > > > > Anyway, I'm adding it to regzbot: > > > > #regzbot introduced: v4.19..v5.15 https://bugzilla.kernel.org/show_bug.cgi?id=217457 > > #regzbot title: bad frame in rt_sigreturn (libc-related?) regression after 5.15 upgrade > > > > Oops, I forgot to add the reporter: > > #regzbot from: Theodor Milkov > > Sorry for inconvenience. > > -- > An old man doll... just what I always wanted! - Clara >