From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7313DC433EF for ; Fri, 24 Sep 2021 08:20:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 447CE61076 for ; Fri, 24 Sep 2021 08:20:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 447CE61076 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:Subject: From:References:Cc:To:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=o6MvPkC3J/9LzaUMN8GO9d8Fl4BvJBfzFhshCeVSiOY=; b=d4rJaC4mh2ay+R+YU/0574OvQe LZqQXILSCihljpnAYWTkTtnjkWT9jqMBw5SgFXgHTVyKo1j8noKU8aW5DK4VJGK9iW2nWE+OSuyZu hJukRBdggGo9jEWOOkAmHN97JHC+YQcxQYXK3XY6qh08unAth9bpHsXEowyoODqKI3kRuVsj3nQnS TGo5xxdggFDeGs5BsM1odHj6khPHnfeYVNnZm5QSu7HGizifHPBsksrjWScVArV9ERgAQk9yCH4AL ohFTrd2xzYGOseuMer4Z8Gpz3In47GYEpR4+8oGnWroU6LIVPOya+hqasS/M/tynJvG1+wKglQdZs W1NVFV+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mTgPF-00DTtd-A3; Fri, 24 Sep 2021 08:18:05 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mTgP5-00DToh-Pm for linux-arm-kernel@lists.infradead.org; Fri, 24 Sep 2021 08:17:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632471473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Hv0DavdoyFzFPDaDYsQZiKG5xroiwpLoHl3JpBTrMIc=; b=PSk0vi48CyylLPcMDGGYHZlYFtSdboVe2vEhb9WFnTa8BRx8ZqlSphqLrBQP2hXxNf5fup I/K7a8zFr0GbYHtWu2z2t1KaMdaxSwYN8MtG6LIJHLL8ZTpDApexjxg5AgxvqkjdhHHa7h qh41xgXNmkJhLvaOwGRps3+4kmg8gl0= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-170-gCDLsQEaPYqdLEXMTSVwNw-1; Fri, 24 Sep 2021 04:17:49 -0400 X-MC-Unique: gCDLsQEaPYqdLEXMTSVwNw-1 Received: by mail-wr1-f69.google.com with SMTP id r15-20020adfce8f000000b0015df1098ccbso7419340wrn.4 for ; Fri, 24 Sep 2021 01:17:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:to:cc:references:from:organization:subject :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=Hv0DavdoyFzFPDaDYsQZiKG5xroiwpLoHl3JpBTrMIc=; b=qIDAxlm47+WsUKSDR/CVM3yZV4rsdbx8qbNBuop5N29qvgYaP0n+WyP3HFu+PKPF8K kgl34oUfNvtHZGhnQVVRLWWiwo+MUs1pm9joGA4jROvOqVrexj8pbNEA3+pVJ6NZRMsR ms1gpwV8qL8j4L/s0b06tZv/7PK8Sy9CDJg79WloWEpoOWkreC0pegeZh9/VJc63X2n0 4POmWCS6vTi5RQ7g3hczMugV0MtoHkKmxJTQ6D5G6YvI0+Hjsa1W8nJgJK757FyuJ9x9 99GxSjiWyWlwHeblxXm83PuUVnAhUpZmUdtcDI7U9/WO3eckgwMWBC8RfyUKR1Cv6Q2E ETeg== X-Gm-Message-State: AOAM530sMFhEYu9mLUtQpbTqfwZ7KaCfWjhCwusaU6k0c6hrmJwRuufx VKiypn0wJei2ofEIgp4YPg7l228DtWT4PRuo+LHd5kmzxvveaddompimmpJlJmbNGVCeQNCOam2 FzroXbR57D3RmwAqqcoWVLX0L93JuwxzKNSU= X-Received: by 2002:a1c:f405:: with SMTP id z5mr671655wma.33.1632471467844; Fri, 24 Sep 2021 01:17:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzrH74aZKFo1LlKFHQ42orrvv8hwQA4S5bh26q5ompTUH91rPUY1MEp2TJJUur1lv+fSj85rg== X-Received: by 2002:a1c:f405:: with SMTP id z5mr671634wma.33.1632471467602; Fri, 24 Sep 2021 01:17:47 -0700 (PDT) Received: from [192.168.3.132] (p5b0c61fc.dip0.t-ipconnect.de. [91.12.97.252]) by smtp.gmail.com with ESMTPSA id f1sm7642302wri.43.2021.09.24.01.17.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Sep 2021 01:17:47 -0700 (PDT) To: Florian Fainelli , Chris Goldsworthy , Catalin Marinas , Will Deacon , Andrew Morton Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sudarshan Rajagopalan , Doug Berger References: <595d09279824faf1f54961cef52b745609b05d97.1632437225.git.quic_cgoldswo@quicinc.com> <6eb8319d-acba-b69a-5db3-5dca9ef426e8@gmail.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [RFC] arm64: mm: update max_pfn after memory hotplug Message-ID: <41789cad-76c6-0ea5-4aa1-3e4a52acff86@redhat.com> Date: Fri, 24 Sep 2021 10:17:46 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <6eb8319d-acba-b69a-5db3-5dca9ef426e8@gmail.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210924_011755_946180_E173DF21 X-CRM114-Status: GOOD ( 36.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 24.09.21 04:47, Florian Fainelli wrote: > > > On 9/23/2021 3:54 PM, Chris Goldsworthy wrote: >> From: Sudarshan Rajagopalan >> >> After new memory blocks have been hotplugged, max_pfn and max_low_pfn >> needs updating to reflect on new PFNs being hot added to system. >> >> Signed-off-by: Sudarshan Rajagopalan >> Signed-off-by: Chris Goldsworthy >> --- >> arch/arm64/mm/mmu.c | 5 +++++ >> 1 file changed, 5 insertions(+) >> >> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c >> index cfd9deb..fd85b51 100644 >> --- a/arch/arm64/mm/mmu.c >> +++ b/arch/arm64/mm/mmu.c >> @@ -1499,6 +1499,11 @@ int arch_add_memory(int nid, u64 start, u64 size, >> if (ret) >> __remove_pgd_mapping(swapper_pg_dir, >> __phys_to_virt(start), size); >> + else { >> + max_pfn = PFN_UP(start + size); >> + max_low_pfn = max_pfn; >> + } > > This is a drive by review, but it got me thinking about your changes a bit: > > - if you raise max_pfn when you hotplug memory, don't you need to lower > it when you hot unplug memory as well? The issue with lowering is that you actually have to do some search to figure out the actual value -- and it's not really worth the trouble. Raising the limit is easy. With memory hotunplug, anybody wanting to take a look at a "struct page" via a pfn has to do a pfn_to_online_page() either way. That will fail if there isn't actually a memmap anymore because the memory has been unplugged. So "max_pfn" is actually rather a hint what maximum pfn to look at, and it can be bigger than it actually is. The a look at the example usage in fs/proc/page.c:kpageflags_read() pfn_to_online_page() will simply fail and stable_page_flags() will indicate a KPF_NOPAGE. Just like we would have a big memory hole now at the end of memory. > > - suppose that you have a platform which maps physical memory into the > CPU's address space at 0x00_4000_0000 (1GB offset) and the kernel boots > with 2GB of DRAM plugged by default. At that point we have not > registered a swiotlb because we have less than 4GB of addressable > physical memory, there is no IOMMU in that system, it's a happy world. > Now assume that we plug an additional 2GB of DRAM into that system > adjacent to the previous 2GB, from 0x00_C0000_0000 through > 0x14_0000_0000, now we have physical addresses above 4GB, but we still > don't have a swiotlb, some of our DMA_BIT_MASK(32) peripherals are going > to be unable to DMA from that hot plugged memory, but they could if we > had a swiotlb. That's why platforms that hotplug memory should indicate the maximum possible PFN via some mechanism during boot. On x86-64 (and IIRC also arm64 now), this is done via the ACPI SRAT. And that's where "max_possible_pfn" and "max_pfn" differ. See drivers/acpi/numa/srat.c:acpi_numa_memory_affinity_init(): max_possible_pfn = max(max_possible_pfn, PFN_UP(end - 1));$ Using max_possible_pfn, the OS can properly setup the swiotlb, even thought it wouldn't currently be required when just looking at max_pfn. I documented that for virtio-mem in https://virtio-mem.gitlab.io/user-guide/user-guide-linux.html "swiotlb and DMA memory". > > - now let's go even further but this is very contrived. Assume that the > firmware has somewhat created a reserved memory region with a 'no-map' > attribute thus indicating it does not want a struct page to be created > for a specific PFN range, is it valid to "blindly" raise max_pfn if that > region were to be at the end of the just hot-plugged memory? no-map means that no direct mapping is to be created, right? We would still have a memmap IIRC, and the pages are PG_reserved. Again, I think this is very similar to just having no-map regions like random memory holes within the existing memory layout. What Chris proposes here is very similar to arch/x86/mm/init_64.c:update_end_of_memory_vars() called during arch_add_memory()->add_pages() on x86-64. -- Thanks, David / dhildenb _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel