From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C647CC001E0 for ; Wed, 16 Aug 2023 09:50:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243513AbjHPJuZ (ORCPT ); Wed, 16 Aug 2023 05:50:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243590AbjHPJt5 (ORCPT ); Wed, 16 Aug 2023 05:49:57 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84654DF for ; Wed, 16 Aug 2023 02:49:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692179348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0OFZQ+OXvyBmwhhAQyMup8CjhNg5bEgDX0bYoSJ7Q74=; b=dC2SrTETPC+oOgeVRPeIzXwzqLIzbkq6QXNqm8toAMvB0hCjyLLDns/9sA1+6oPPqXgseA dN9Zx6jh/2PWD+JPjS3EifHdr2uQDyS4u2Ne9XJ/D6Zr79jvH06cJf+lCDFLJfjIhTKabe kmCn+PQdB86qBvEGZINY/ux5qwFJLJE= Received: from mail-lj1-f198.google.com (mail-lj1-f198.google.com [209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-558-Jb-BAB34MReGLPMkb3GZng-1; Wed, 16 Aug 2023 05:49:07 -0400 X-MC-Unique: Jb-BAB34MReGLPMkb3GZng-1 Received: by mail-lj1-f198.google.com with SMTP id 38308e7fff4ca-2b9c548bc66so63189081fa.1 for ; Wed, 16 Aug 2023 02:49:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692179345; x=1692784145; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0OFZQ+OXvyBmwhhAQyMup8CjhNg5bEgDX0bYoSJ7Q74=; b=TMne4XCYGCTmZ9LLtzWggtGFIjelGiVbYuUKaUlvqzl1CaYWej+xT2p3apbk+/yiaD OSYjM8dTIEJQeZwHaYaVg1x+doa8+NzxjLx718cryw0eUW69v98K3Xu9JDt71jFqIYna ndKTKecIzacUWZFVnzth/gK7aM+nHa+cqBUmmBaM+Knm30hbUowHDQM+87i4yZJAcN1R La3otDpI1LVRPhwhiJ91sCugn5Z0kCTdrnWfJwzGRpKm/Zvt3UQ4Lt0EOS/x1AnaZECM 016JGF4pXK26BrUPLZ5vBY8lNTDZz59UYvxFP1qYR44LHXP//4sr9n99uB9o8h0vD1PY HAMw== X-Gm-Message-State: AOJu0Yyq2iY1BWF5ofKMFQwjJkD6P7izcdUSi8YaCRe8r9IXuXcl6PsK kqFH4XvNKFMTZIXsM7Z85XaVoYSMyuLNG33YEOMzgC88i9rqjimO25eeZuIF4o7MKXAIp9NySnm pn2A8MZcnfRHJZUBDe2npsicr X-Received: by 2002:a2e:b285:0:b0:2b4:5cad:f246 with SMTP id 5-20020a2eb285000000b002b45cadf246mr1107452ljx.7.1692179345766; Wed, 16 Aug 2023 02:49:05 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFAcVWmuJNfrQc1yoLpfalLlUny5Yyp2MgsmtGtaEdKirncc0n7pb9TlVylBlmWwHsFeOVlgw== X-Received: by 2002:a2e:b285:0:b0:2b4:5cad:f246 with SMTP id 5-20020a2eb285000000b002b45cadf246mr1107428ljx.7.1692179345278; Wed, 16 Aug 2023 02:49:05 -0700 (PDT) Received: from ?IPV6:2003:cb:c74b:8b00:5520:fa3c:c527:592f? (p200300cbc74b8b005520fa3cc527592f.dip0.t-ipconnect.de. [2003:cb:c74b:8b00:5520:fa3c:c527:592f]) by smtp.gmail.com with ESMTPSA id o13-20020a05600c378d00b003fe2de3f94fsm20525323wmr.12.2023.08.16.02.49.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 16 Aug 2023 02:49:04 -0700 (PDT) Message-ID: Date: Wed, 16 Aug 2023 11:49:03 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Content-Language: en-US To: Yan Zhao Cc: John Hubbard , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, pbonzini@redhat.com, seanjc@google.com, mike.kravetz@oracle.com, apopple@nvidia.com, jgg@nvidia.com, rppt@kernel.org, akpm@linux-foundation.org, kevin.tian@intel.com, Mel Gorman References: <20230810085636.25914-1-yan.y.zhao@intel.com> <41a893e1-f2e7-23f4-cad2-d5c353a336a3@redhat.com> <6b48a161-257b-a02b-c483-87c04b655635@redhat.com> <1ad2c33d-95e1-49ec-acd2-ac02b506974e@nvidia.com> <846e9117-1f79-a5e0-1b14-3dba91ab8033@redhat.com> <4271b91c-90b7-4b48-b761-b4535b2ae9b7@nvidia.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in a VM In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 16.08.23 11:06, Yan Zhao wrote: > On Wed, Aug 16, 2023 at 09:43:40AM +0200, David Hildenbrand wrote: >> On 15.08.23 04:34, John Hubbard wrote: >>> On 8/14/23 02:09, Yan Zhao wrote: >>> ... >>>>> hmm_range_fault()-based memory management in particular might benefit >>>>> from having NUMA balancing disabled entirely for the memremap_pages() >>>>> region, come to think of it. That seems relatively easy and clean at >>>>> first glance anyway. >>>>> >>>>> For other regions (allocated by the device driver), a per-VMA flag >>>>> seems about right: VM_NO_NUMA_BALANCING ? >>>>> >>>> Thanks a lot for those good suggestions! >>>> For VMs, when could a per-VMA flag be set? >>>> Might be hard in mmap() in QEMU because a VMA may not be used for DMA until >>>> after it's mapped into VFIO. >>>> Then, should VFIO set this flag on after it maps a range? >>>> Could this flag be unset after device hot-unplug? >>>> >>> >>> I'm hoping someone who thinks about VMs and VFIO often can chime in. >> >> At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, >> using madvise). >> >> BUT, I do wonder what value there would be for autonuma to still be active > Currently MADV_* is up to 25 > #define MADV_COLLAPSE 25, > while madvise behavior is of type "int". So it's ok. > > But vma->vm_flags is of "unsigned long", so it's full at least on 32bit platform. I remember there were discussions to increase it also for 32bit. If that's required, we might want to go down that path. But do 32bit architectures even care about NUMA hinting? If not, just ignore them ... > >> for the remainder of the hypervisor. If there is none, a prctl() would be >> better. > Add a new field in "struct vma_numab_state" in vma, and use prctl() to > update this field? Rather a global toggle per MM, no need to update individual VMAs -- if we go down that prctl() path. No need to consume more memory for VMAs. [...] >> We already do have a mechanism in QEMU to get notified when longterm-pinning >> in the kernel might happen (and, therefore, MADV_DONTNEED must not be used): >> * ram_block_discard_disable() >> * ram_block_uncoordinated_discard_disable() > Looks this ram_block_discard allow/disallow state is global rather than per-VMA > in QEMU. Yes. Once you transition into "discard of any kind disabled", you can go over all guest memory VMAs (RAMBlock) and issue an madvise() for them. (or alternatively, do the prctl() once ) We'll also have to handle new guest memory being created afterwards, but that is easy. Once we transition to "no discarding disabled", you can go over all guest memory VMAs (RAMBlock) and issue an madvise() for them again (or alternatively, do the prctl() once). > So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and > it's up to the user space to choose between per-VMA and complex way or > global and simpler way? QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting. -- Cheers, David / dhildenb