From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joel Fernandes Subject: Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions Date: Fri, 12 Oct 2018 09:57:19 -0700 Message-ID: <20181012165719.GE223066@joelaf.mtv.corp.google.com> References: <20181012013756.11285-1-joel@joelfernandes.org> <20181012013756.11285-2-joel@joelfernandes.org> <20181012113056.gxhcbrqyu7k7xnyv@kshutemo-mobl1> <20181012125046.GA170912@joelaf.mtv.corp.google.com> <20181012131946.zoab2lpfmrycmuju@kshutemo-mobl1> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iSLo6yn953RzmEcjKApVIHFLWIftCd2IlkuT4QhGmzA=; b=asSHGIPPspdg4z i4/55DK9qfrw4ZYTbgkdfHEOZhTCN69iF12cw268dFasq56K5anEy6oZV7n+0vOWkapuXyuVsR+WX maXkk0ilk7oK8ecb892tfhfBcStV7bMWoZ9KlyUaS1D/wc3860Z4/p7RCN2xFvX4xfRhuI3pGIOfJ HBvt3ErLhm+E/nkfM93FGO8kZ9FOHAw3joq/5Up096vaMlbSKb+/LDFu+0Iwn70pdGyThfbEZzFme zGoe4+FyzQ9sEpxY+6oZDV4szBOqHvjD1xdoGje5SOZ3h13vEw5wQhkKqXwvXCTbNAjsB+3qWhUxo +i4oZKSe/iHbW9hB8ilQ==; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=uVlHzOjCJt1s37XP7P3Q38uoyVSxQUIUy8JlaGTffKg=; b=cXLWIF0psQEeOaff/zCLiIOyVLAunEJjwQpMHAjktpY4IMZnzm2ghp8KflvQg70gdF EQ3wGdn0DuWTTsMR3/qYn6+Mr2xvpeoMKUhr8DIN5b3h412YRn5qTL3lOpo0fcgGwZVL ycFj3Jxu7+NinK3bZZEUua7aW3602iGTxtcqw= Content-Disposition: inline In-Reply-To: <20181012131946.zoab2lpfmrycmuju@kshutemo-mobl1> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+glpr-linux-riscv=m.gmane.org@lists.infradead.org To: "Kirill A. Shutemov" Cc: linux-mips@linux-mips.org, Rich Felker , linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Dave Hansen , Will Deacon , mhocko@kernel.org, linux-mm@kvack.org, lokeshgidra@google.com, sparclinux@vger.kernel.org, linux-riscv@lists.infradead.org, elfring@users.sourceforge.net, Jonas Bonn , linux-s390@vger.kernel.org, dancol@google.com, Yoshinori Sato , linux-xtensa@linux-xtensa.org, linux-hexagon@vger.kernel.org, Helge Deller , "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" , hughd@google.com, "James E.J. Bottomley" , kasan-dev@googlegroups.com, kvmarm@lists.cs.columbia.edu, Ingo Molnar , Geert On Fri, Oct 12, 2018 at 04:19:46PM +0300, Kirill A. Shutemov wrote: > On Fri, Oct 12, 2018 at 05:50:46AM -0700, Joel Fernandes wrote: > > On Fri, Oct 12, 2018 at 02:30:56PM +0300, Kirill A. Shutemov wrote: > > > On Thu, Oct 11, 2018 at 06:37:56PM -0700, Joel Fernandes (Google) wrote: > > > > Android needs to mremap large regions of memory during memory management > > > > related operations. The mremap system call can be really slow if THP is > > > > not enabled. The bottleneck is move_page_tables, which is copying each > > > > pte at a time, and can be really slow across a large map. Turning on THP > > > > may not be a viable option, and is not for us. This patch speeds up the > > > > performance for non-THP system by copying at the PMD level when possible. > > > > > > > > The speed up is three orders of magnitude. On a 1GB mremap, the mremap > > > > completion times drops from 160-250 millesconds to 380-400 microseconds. > > > > > > > > Before: > > > > Total mremap time for 1GB data: 242321014 nanoseconds. > > > > Total mremap time for 1GB data: 196842467 nanoseconds. > > > > Total mremap time for 1GB data: 167051162 nanoseconds. > > > > > > > > After: > > > > Total mremap time for 1GB data: 385781 nanoseconds. > > > > Total mremap time for 1GB data: 388959 nanoseconds. > > > > Total mremap time for 1GB data: 402813 nanoseconds. > > > > > > > > Incase THP is enabled, the optimization is skipped. I also flush the > > > > tlb every time we do this optimization since I couldn't find a way to > > > > determine if the low-level PTEs are dirty. It is seen that the cost of > > > > doing so is not much compared the improvement, on both x86-64 and arm64. > > > > > > I looked into the code more and noticed move_pte() helper called from > > > move_ptes(). It changes PTE entry to suite new address. > > > > > > It is only defined in non-trivial way on Sparc. I don't know much about > > > Sparc and it's hard for me to say if the optimization will break anything > > > there. > > > > Sparc's move_pte seems to be flushing the D-cache to prevent aliasing. It is > > not modifying the PTE itself AFAICS: > > > > #ifdef DCACHE_ALIASING_POSSIBLE > > #define __HAVE_ARCH_MOVE_PTE > > #define move_pte(pte, prot, old_addr, new_addr) \ > > ({ \ > > pte_t newpte = (pte); \ > > if (tlb_type != hypervisor && pte_present(pte)) { \ > > unsigned long this_pfn = pte_pfn(pte); \ > > \ > > if (pfn_valid(this_pfn) && \ > > (((old_addr) ^ (new_addr)) & (1 << 13))) \ > > flush_dcache_page_all(current->mm, \ > > pfn_to_page(this_pfn)); \ > > } \ > > newpte; \ > > }) > > #endif > > > > If its an issue, then how do transparent huge pages work on Sparc? I don't > > see the huge page code (move_huge_pages) during mremap doing anything special > > for Sparc architecture when moving PMDs.. > > My *guess* is that it will work fine on Sparc as it apprarently it only > cares about change in bit 13 of virtual address. It will never happen for > huge pages or when PTE page tables move. > > But I just realized that the problem is bigger: since we pass new_addr to > the set_pte_at() we would need to audit all implementations that they are > safe with just moving PTE page table. > > I would rather go with per-architecture enabling. It's much safer. I'm Ok with the per-arch enabling, I agree its safer. So I should be adding a a new __HAVE_ARCH_MOVE_PMD right, or did you have a better name for that? Also, do you feel we should still need to remove the address argument from set_pte_alloc? Or should we leave that alone if we do per-arch? I figure I spent a bunch of time on that already anyway, and its a clean up anyway, so may as well do it. But perhaps that "pte_alloc cleanup" can then be a separate patch independent of this series? > > Also, do we not flush the caches from any path when we munmap address space? > > We do call do_munmap on the old mapping from mremap after moving to the new one. > > Are you sure about that? It can be hided deeper in architecture-specific > code. I am sure we do call do_munmap, I was asking if we flush the caches as well. If we're enabling this per architecture, then I guess it does not matter for the purposes of this patch. thanks, - Joel