From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Subject: Re: another pmem variant V2 Date: Wed, 1 Apr 2015 09:26:08 +0200 Message-ID: <20150401072608.GA24312@lst.de> References: <1427358764-6126-1-git-send-email-hch@lst.de> <94D0CD8314A33A4D9D801C0FE68B40295A853392@G9W0745.americas.hpqcorp.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Christoph Hellwig , "linux-nvdimm@ml01.01.org" , "linux-fsdevel@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "x86@kernel.org" , "ross.zwisler@linux.intel.com" , "axboe@kernel.dk" , "boaz@plexistor.com" , "Kani, Toshimitsu" To: "Elliott, Robert (Server Storage)" Return-path: Content-Disposition: inline In-Reply-To: <94D0CD8314A33A4D9D801C0FE68B40295A853392@G9W0745.americas.hpqcorp.net> Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org On Tue, Mar 31, 2015 at 10:11:29PM +0000, Elliott, Robert (Server Storage) wrote: > I used fio to test 4 KiB random read and write IOPS > on a 2-socket x86 DDR4 system. With various cache attributes: > > attr read write notes > ---- ---- ----- ----- > UC 37 K 21 K ioremap_nocache > WB 3.6 M 2.5 M ioremap > WC 764 K 3.7 M ioremap_wc > WT ioremap_wt > > So, although UC and WT are the only modes certain to be safe, > the V1 default of UC provides abysmal performance - worse than > a consumer-class SATA SSD. It doesn't look quite as bad on my setup, but performance is fairly bad here as well. > A solution for x86 is to use the MOVNTI instruction in WB > mode. This non-temporal hint uses a buffer like the write > combining buffer, not filling the cache and not stopping > everything in the CPU. The kernel function __copy_from_user() > uses that instruction (with SFENCE at the end) - see > arch/x86/lib/copy_user_nocache_64.S. > > If I made the change from memcpy() to __copy_from_user() > correctly, that results in: > > attr read write notes > ---- ---- ----- ----- > WB w/NTI 2.4 M 2.6 M __copy_from_user() > WC w/NTI 3.2 M 2.1 M __copy_from_user() That looks a lot better. It doesn't help us with a pmem device mapped directly into userspace using mmap with the DAX infrastructure, though. Note when we want to move to non-temporal copies we'll need to add a new prototype, as __copy_from_user isn't guaranteed to use these, and it is defined to only work on user addresses. That doesn't matter on x86 but would blow up on say sparc or s390.