From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7F98C433E0 for ; Thu, 28 Jan 2021 15:04:40 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 57E7464DF9 for ; Thu, 28 Jan 2021 15:04:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 57E7464DF9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:Subject: From:References:To:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=EbwdNS3Dp11x5ImpDPST7JHmR/LFVj2TWuJ8QbFlo9s=; b=Ftk1TKgwcNFwMTOeke6PEb+Tm 4l0Ux5QK2qQFy/khH8FnziN3r5tz4LzCgobvuJqGooitqXk15fQPX4EUgc4kx6d8sgX2LxpU02lzA ArPOAipJh4a3E63tPFbynttJCgggbcU/6F87qimJ/Bhf/VSUkBe1tgEeSN7zJFHKO8bzXdCdQ4SiB bfPNV/LmUMDgDMyRapfSca+PHsHgHz89njgxB20RgFaO3WuIp0Nwjq8sI4w68mBLMkFX2c2UIfuSC wo1QVZL016SfNEXWX2GsM1HUjJvHjQekDQ95IeebEi9NyvQup25qnu9AsMqBZMQF0QDnJ7IGMsvSH iUiFph84g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l58pR-0002OV-VN; Thu, 28 Jan 2021 15:03:26 +0000 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l58pO-0002Ms-D2 for linux-arm-kernel@lists.infradead.org; Thu, 28 Jan 2021 15:03:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611846201; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ChbOkEgbATFM37SwTaCF7y0/gcttE6lmGqqCimF4GA8=; b=NakydoJ9h1ceN5CE7BYMNXJyGgNZ2zTJUZU2q07QU+GQYwZH6geO6/Dl5MOOlUdcjXbzQ/ wAkz3zxVpyu5VfbsFrw8xBRGr/IdxqadoSIcgxNJemO4wTMY4Q2kTKhn6HGKx7NvYYUnLW PHaIwAerF7vdWJFev7rQtzelOf7i52c= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-290-h47oyd7PPFO5_XSp3oVzYw-1; Thu, 28 Jan 2021 10:03:16 -0500 X-MC-Unique: h47oyd7PPFO5_XSp3oVzYw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 21BC210052FE; Thu, 28 Jan 2021 15:03:13 +0000 (UTC) Received: from [10.36.113.207] (ovpn-113-207.ams2.redhat.com [10.36.113.207]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2A1845D720; Thu, 28 Jan 2021 15:03:08 +0000 (UTC) To: Pavel Tatashin References: <8c2b75fe-a3e5-8eff-7f37-5d23c7ad9742@redhat.com> <94797c92-cd90-8a65-b879-0bb5f12b9fc5@redhat.com> From: David Hildenbrand Organization: Red Hat GmbH Subject: Re: dax alignment problem on arm64 (and other achitectures) Message-ID: Date: Thu, 28 Jan 2021 16:03:07 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210128_100322_485191_79BA5A34 X-CRM114-Status: GOOD ( 21.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Michal Hocko , Jason Gunthorpe , Marc Zyngier , Will Deacon , LKML , James Morris , linux-mm , Tyler Hicks , James Morse , Linux ARM , Joonsoo Kim , Andrew Morton , Dan Williams , Vlastimil Babka , Oscar Salvador Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org >> One issue usually is that often firmware can allocate from available >> system RAM and/or modify/initialize it. I assume you're running some >> custom firmware :) > > We have a special firmware that does not touch the last 2G of physical > memory for its allocations :) > Fancy :) [...] >> Personally, I think the future is 4k, especially for smaller machines. >> (also, imagine right now how many 512MB THP you can actually use in your >> 8GB VM ..., simply not suitable for small machines). > > Um, this is not really about 512THP. Yes, this is smaller machine, but > performance is very important to us. Boot budget for the kernel is > under half a second. With 64K we save 0.2s 0.35s vs 0.55s. This is > because fewer struct pages need to be initialized. Also, fewer TLB > misses, and 3-level page tables add up as performance benefits. > > For larger servers 64K pages make total sense: Less memory is wasted as metdata. Yes, indeed, for very large servers it might make sense in that regard. However, once we can eventually free vmemmap of hugetlbfs things could change; assuming user space will be consuming huge pages (which large machines better be doing ... databases, hypervisors ... ). Also, some hypervisors try allocating the memmap completely ... but I consider that rather a special case. Personally, I consider being able to use THP/huge pages more important than having 64k base pages and saving some TLB space there. Also, with 64k you have other drawbacks: for example, each stack, each TLS for threads in applications suddenly consumes 16 times more memory as "minimum". Optimizing boot time/memmap initialization further is certainly an interesting topic. Anyhow, you know your use case best, just sharing my thoughts :) [...] >>> >>> Right, but I do not think it is possible to do for dax devices (as of >>> right now). I assume, it contains information about what kind of >>> device it is: devdax, fsdax, sector, uuid etc. >>> See [1] namespaces tabel. It contains summary of pmem devices types, >>> and which of them have label (all except for raw). >> >> Interesting, I wonder if the label is really required to get this >> special use case running. I mean, all you want is to have dax/kmem >> expose the whole thing as system RAM. You don't want to lose even 2MB if >> it's just for the sake of unnecessary metadata - this is not a real >> device, it's "fake" already. > > Hm, would not it essentially mean allowing memory hot-plug for raw > pmem devices? Something like create mmap, and hot-add raw pmem? Theoretically yes, but I have no idea if that would make sense for real "raw pmem" as well. Hope some of the pmem/nvdimm experts can clarify what's possible and what's not :) -- Thanks, David / dhildenb _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD321C433E6 for ; Thu, 28 Jan 2021 15:03:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6754D64DD9 for ; Thu, 28 Jan 2021 15:03:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6754D64DD9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD14E6B006C; Thu, 28 Jan 2021 10:03:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C82EA6B0070; Thu, 28 Jan 2021 10:03:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B989F6B0072; Thu, 28 Jan 2021 10:03:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id A465B6B006C for ; Thu, 28 Jan 2021 10:03:22 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 29148181AEF2A for ; Thu, 28 Jan 2021 15:03:22 +0000 (UTC) X-FDA: 77755502244.10.legs81_330b475275a0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id EC19716A07F for ; Thu, 28 Jan 2021 15:03:21 +0000 (UTC) X-HE-Tag: legs81_330b475275a0 X-Filterd-Recvd-Size: 5947 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 Jan 2021 15:03:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611846200; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ChbOkEgbATFM37SwTaCF7y0/gcttE6lmGqqCimF4GA8=; b=QBSv4uI+PVhX6wGnJs//jU3f/kmMy0rBSyTdIPECEoniHpnUip53ndDByc8QkMu8zQCt7S HYFOi4noFSLcFijWd3cYP7eo6+BG1rzX2jdMTJFNuL/JesmbPQ+goN7x3EchM1f7xYnmTG 2NRy08+i/7BQVCzk/aHMnBLibyRg+EY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-290-h47oyd7PPFO5_XSp3oVzYw-1; Thu, 28 Jan 2021 10:03:16 -0500 X-MC-Unique: h47oyd7PPFO5_XSp3oVzYw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 21BC210052FE; Thu, 28 Jan 2021 15:03:13 +0000 (UTC) Received: from [10.36.113.207] (ovpn-113-207.ams2.redhat.com [10.36.113.207]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2A1845D720; Thu, 28 Jan 2021 15:03:08 +0000 (UTC) To: Pavel Tatashin Cc: linux-mm , LKML , Sasha Levin , Tyler Hicks , Andrew Morton , Dan Williams , Michal Hocko , Oscar Salvador , Vlastimil Babka , Joonsoo Kim , Jason Gunthorpe , Marc Zyngier , Linux ARM , Will Deacon , James Morse , James Morris References: <8c2b75fe-a3e5-8eff-7f37-5d23c7ad9742@redhat.com> <94797c92-cd90-8a65-b879-0bb5f12b9fc5@redhat.com> From: David Hildenbrand Organization: Red Hat GmbH Subject: Re: dax alignment problem on arm64 (and other achitectures) Message-ID: Date: Thu, 28 Jan 2021 16:03:07 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >> One issue usually is that often firmware can allocate from available >> system RAM and/or modify/initialize it. I assume you're running some >> custom firmware :) >=20 > We have a special firmware that does not touch the last 2G of physical > memory for its allocations :) >=20 Fancy :) [...] >> Personally, I think the future is 4k, especially for smaller machines. >> (also, imagine right now how many 512MB THP you can actually use in yo= ur >> 8GB VM ..., simply not suitable for small machines). >=20 > Um, this is not really about 512THP. Yes, this is smaller machine, but > performance is very important to us. Boot budget for the kernel is > under half a second. With 64K we save 0.2s 0.35s vs 0.55s. This is > because fewer struct pages need to be initialized. Also, fewer TLB > misses, and 3-level page tables add up as performance benefits. > > For larger servers 64K pages make total sense: Less memory is wasted as= metdata. Yes, indeed, for very large servers it might make sense in that regard.=20 However, once we can eventually free vmemmap of hugetlbfs things could=20 change; assuming user space will be consuming huge pages (which large=20 machines better be doing ... databases, hypervisors ... ). Also, some hypervisors try allocating the memmap completely ... but I=20 consider that rather a special case. Personally, I consider being able to use THP/huge pages more important=20 than having 64k base pages and saving some TLB space there. Also, with=20 64k you have other drawbacks: for example, each stack, each TLS for=20 threads in applications suddenly consumes 16 times more memory as "minimu= m". Optimizing boot time/memmap initialization further is certainly an=20 interesting topic. Anyhow, you know your use case best, just sharing my thoughts :) [...] >>> >>> Right, but I do not think it is possible to do for dax devices (as of >>> right now). I assume, it contains information about what kind of >>> device it is: devdax, fsdax, sector, uuid etc. >>> See [1] namespaces tabel. It contains summary of pmem devices types, >>> and which of them have label (all except for raw). >> >> Interesting, I wonder if the label is really required to get this >> special use case running. I mean, all you want is to have dax/kmem >> expose the whole thing as system RAM. You don't want to lose even 2MB = if >> it's just for the sake of unnecessary metadata - this is not a real >> device, it's "fake" already. >=20 > Hm, would not it essentially mean allowing memory hot-plug for raw > pmem devices? Something like create mmap, and hot-add raw pmem? Theoretically yes, but I have no idea if that would make sense for real=20 "raw pmem" as well. Hope some of the pmem/nvdimm experts can clarify=20 what's possible and what's not :) --=20 Thanks, David / dhildenb