From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99A7CC55178 for ; Tue, 3 Nov 2020 14:04:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 46F5022280 for ; Tue, 3 Nov 2020 14:04:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="G+BmjeiH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729486AbgKCOEx (ORCPT ); Tue, 3 Nov 2020 09:04:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:35105 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729440AbgKCOEF (ORCPT ); Tue, 3 Nov 2020 09:04:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604412243; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=IsejNI8EYSmowoiAvOvFY33zKF4jozyNUSQ9LBHEIew=; b=G+BmjeiHfQkXjo4My/zO9tdhf0PVaW6LzA6xieVKsT5aLa3SQ+ShQaVSrO9xBSVu7t3APa uBxUBvfLqOmYlO6YTGOT195Iev6lE3DfIyJx5UZxqf+SxQ+wHN+ULxm3QVPctVECVC2ubb ePwmjgOqrpCbmh3SGWKBKM2gK7J2ZpQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-595-cvTYnFueNbGheckDPJMuxw-1; Tue, 03 Nov 2020 09:03:58 -0500 X-MC-Unique: cvTYnFueNbGheckDPJMuxw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7A2E3107465A; Tue, 3 Nov 2020 14:03:56 +0000 (UTC) Received: from localhost (ovpn-12-165.pek2.redhat.com [10.72.12.165]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5383F5C230; Tue, 3 Nov 2020 14:03:55 +0000 (UTC) Date: Tue, 3 Nov 2020 22:03:52 +0800 From: "bhe@redhat.com" To: Rahul Gopakumar Cc: "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "akpm@linux-foundation.org" , "natechancellor@gmail.com" , "ndesaulniers@google.com" , "clang-built-linux@googlegroups.com" , "rostedt@goodmis.org" , Rajender M , Yiu Cho Lau , Peter Jonasson , Venkatesh Rajaram Subject: Re: Performance regressions in "boot_time" tests in Linux 5.8 Kernel Message-ID: <20201103140352.GB3177@MiWiFi-R3L-srv> References: <20201013131735.GL25604@MiWiFi-R3L-srv> <20201020151814.GU25604@MiWiFi-R3L-srv> <20201022040440.GX25604@MiWiFi-R3L-srv> <20201102143035.GA3177@MiWiFi-R3L-srv> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/03/20 at 12:34pm, Rahul Gopakumar wrote: > >> So, you mean with the draft patch applied, the initial performance > regression goes away, just many page corruption errors with call trace > are seen, right? > > Yes, that's right. > > >> And the performance regression is about 2sec delay in > your system? > > The delay due to this new page corruption issue is about > 3 secs. > > Here is the summary > > * Initial problem - 2 secs > * Draft patch - Fixes initial problem (recovers 2 secs) but > brings in new page corruption issue (3 secs) > > >> Could you tell how you setup vmware VM so that I can ask our QA for > help to create a vmware VM for me to test? > > * Use vSphere ESXi 6.7 or 7.0 GA. > * Create VM using vSphere Web Client and specify 1TB VM Memory. > * Install RHEL 8.1, that's the guest used in this test. OK, I see. The draft patch fix the original issue, seems some boundary of memory region is not handled correctly. Thanks for confirmation. The memory layout is important in this case. Not sure if making a VM gesut as you suggested can also create a system with below memory layout. [ 0.008842] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] [ 0.008842] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff] [ 0.008843] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x55ffffffff] [ 0.008844] ACPI: SRAT: Node 1 PXM 1 [mem 0x5600000000-0xaaffffffff] [ 0.008844] ACPI: SRAT: Node 2 PXM 2 [mem 0xab00000000-0xfcffffffff] [ 0.008845] ACPI: SRAT: Node 2 PXM 2 [mem 0x10000000000-0x1033fffffff] > > With draft patch, you should be able to reproduce the issue. > Let me know if you need more details.