From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ZenIV.linux.org.uk (zeniv.linux.org.uk [195.92.253.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 6EA47817DF for ; Tue, 3 Jan 2017 15:22:18 -0800 (PST) Date: Tue, 3 Jan 2017 23:22:06 +0000 From: Al Viro Subject: Re: [RFC] memcpy_nocache() and memcpy_writethrough() Message-ID: <20170103232206.GD1555@ZenIV.linux.org.uk> References: <5818A5C8.6040300@plexistor.com> <20161228234321.GA27417@ZenIV.linux.org.uk> <20161230035252.GV1555@ZenIV.linux.org.uk> <20161231022558.GW1555@ZenIV.linux.org.uk> <20170102050927.GY1555@ZenIV.linux.org.uk> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Dan Williams Cc: "linux-nvdimm@lists.01.org" , "Moreno," Oliver" , "x86@kernel.org, " , linux-kernel@vger.kernel.org" , Ingo Molnar , "H. Peter Anvin" , Thomas Gleixner , Linus Torvalds , "boylston@burromesa.net" List-ID: On Tue, Jan 03, 2017 at 01:14:11PM -0800, Dan Williams wrote: > Robert was describing the overall flow / mechanics, but I think it is > easier to visualize the sfence as a flush command sent to a disk > device with a volatile cache. In fact, that's how we implemented it in > the pmem block device driver. The pmem block device registers itself > as requiring REQ_FLUSH to be sent to persist writes. The driver issues > sfence on the assumption that all writes to pmem have either bypassed > the cache with movnt, or are scheduled for write-back via one of the > flush instructions (clflush, clwb, or clflushopt). *blink* 1) memcpy_to_pmem() seems to rely upon the __copy_from_user_nocache() having only used movnt; it does not attempt clwb at all. 2) __copy_from_user_nocache() for short copies does not use movnt at all. In that case neither sfence nor clwb is issued. 3) it uses movnt only for part of copying in case of misaligned copy; No clwb is issued, but sfence *is* - at the very end in 64bit case, between movnt and copying the tail - in 32bit one. Incidentally, while 64bit case takes care to align the destination for movnt part, 32bit one does not. How much of the above is broken and what do the callers rely upon? In particular, is that sfence the right thing for pmem usecases? _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1761969AbdACXX3 (ORCPT ); Tue, 3 Jan 2017 18:23:29 -0500 Received: from zeniv.linux.org.uk ([195.92.253.2]:44570 "EHLO ZenIV.linux.org.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1761899AbdACXXK (ORCPT ); Tue, 3 Jan 2017 18:23:10 -0500 Date: Tue, 3 Jan 2017 23:22:06 +0000 From: Al Viro To: Dan Williams Cc: "Elliott, Robert (Persistent Memory)" , Boaz Harrosh , "linux-nvdimm@lists.01.org" , "Moreno, Oliver" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , Ingo Molnar , "H. Peter Anvin" , Thomas Gleixner , "boylston@burromesa.net" , Linus Torvalds Subject: Re: [RFC] memcpy_nocache() and memcpy_writethrough() Message-ID: <20170103232206.GD1555@ZenIV.linux.org.uk> References: <5818A5C8.6040300@plexistor.com> <20161228234321.GA27417@ZenIV.linux.org.uk> <20161230035252.GV1555@ZenIV.linux.org.uk> <20161231022558.GW1555@ZenIV.linux.org.uk> <20170102050927.GY1555@ZenIV.linux.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.1 (2016-10-04) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jan 03, 2017 at 01:14:11PM -0800, Dan Williams wrote: > Robert was describing the overall flow / mechanics, but I think it is > easier to visualize the sfence as a flush command sent to a disk > device with a volatile cache. In fact, that's how we implemented it in > the pmem block device driver. The pmem block device registers itself > as requiring REQ_FLUSH to be sent to persist writes. The driver issues > sfence on the assumption that all writes to pmem have either bypassed > the cache with movnt, or are scheduled for write-back via one of the > flush instructions (clflush, clwb, or clflushopt). *blink* 1) memcpy_to_pmem() seems to rely upon the __copy_from_user_nocache() having only used movnt; it does not attempt clwb at all. 2) __copy_from_user_nocache() for short copies does not use movnt at all. In that case neither sfence nor clwb is issued. 3) it uses movnt only for part of copying in case of misaligned copy; No clwb is issued, but sfence *is* - at the very end in 64bit case, between movnt and copying the tail - in 32bit one. Incidentally, while 64bit case takes care to align the destination for movnt part, 32bit one does not. How much of the above is broken and what do the callers rely upon? In particular, is that sfence the right thing for pmem usecases?