[PATCH] x86: Add kerneldoc for pcommit_sfence()

[PATCH] x86: Add kerneldoc for pcommit_sfence()

[Ross Zwisler]

Add kerneldoc comments for pcommit_sfence() describing the purpose of
the pcommit instruction and demonstrating the usage of that instruction.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Cc: H Peter Anvin <h.peter.anvin@intel.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Borislav Petkov <bp@alien8.de>
---
 arch/x86/include/asm/special_insns.h | 37 ++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
index aeb4666e0c0a..1ae81757c05b 100644
--- a/arch/x86/include/asm/special_insns.h
+++ b/arch/x86/include/asm/special_insns.h
@@ -215,6 +215,43 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 
+/**
+ * pcommit_sfence() - persistent commit and fence
+ *
+ * The pcommit instruction ensures that data that has been flushed from the
+ * processor's cache hierarchy with clwb, clflushopt or clflush is accepted to
+ * memory and is durable on the DIMM.  The primary use case for this is
+ * persistent memory.
+ *
+ * This function shows how to properly use clwb/clflushopt/clflush and pcommit
+ * with appropriate fencing:
+ *
+ * void flush_and_commit_buffer(void *vaddr, unsigned int size)
+ * {
+ *         unsigned long clflush_mask = boot_cpu_data.x86_clflush_size - 1;
+ *         char *vend = (char *)vaddr + size;
+ *         char *p;
+ *
+ *         for (p = (char *)((unsigned long)vaddr & ~clflush_mask);
+ *              p < vend; p += boot_cpu_data.x86_clflush_size)
+ *                 clwb(p);
+ *
+ *         // sfence to order clwb/clflushopt/clflush cache flushes
+ *         // mfence via mb() also works
+ *         wmb();
+ *
+ *         // pcommit and the required sfence for ordering
+ *         pcommit_sfence();
+ * }
+ *
+ * After this function completes the data pointed to by vaddr is has been
+ * accepted to memory and will be durable if the vaddr points to persistent
+ * memory.
+ *
+ * Pcommit must always be ordered by an mfence or sfence, so to help simplify
+ * things we include both the pcommit and the required sfence in the
+ * alternatives generated by pcommit_sfence().
+ */
 static inline void pcommit_sfence(void)
 {
 	alternative(ASM_NOP7,
-- 
1.9.3

Re: [PATCH] x86: Add kerneldoc for pcommit_sfence()

[Borislav Petkov]

On Wed, Mar 11, 2015 at 12:19:21PM -0600, Ross Zwisler wrote:
> Add kerneldoc comments for pcommit_sfence() describing the purpose of
> the pcommit instruction and demonstrating the usage of that instruction.
> 
> Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Cc: H Peter Anvin <h.peter.anvin@intel.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Borislav Petkov <bp@alien8.de>
> ---
>  arch/x86/include/asm/special_insns.h | 37 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 37 insertions(+)
> 
> diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
> index aeb4666e0c0a..1ae81757c05b 100644
> --- a/arch/x86/include/asm/special_insns.h
> +++ b/arch/x86/include/asm/special_insns.h
> @@ -215,6 +215,43 @@ static inline void clwb(volatile void *__p)
>  		: [pax] "a" (p));
>  }
>  
> +/**
> + * pcommit_sfence() - persistent commit and fence
> + *
> + * The pcommit instruction ensures that data that has been flushed from the
> + * processor's cache hierarchy with clwb, clflushopt or clflush is accepted to
> + * memory and is durable on the DIMM.  The primary use case for this is
> + * persistent memory.
> + *
> + * This function shows how to properly use clwb/clflushopt/clflush and pcommit
> + * with appropriate fencing:
> + *
> + * void flush_and_commit_buffer(void *vaddr, unsigned int size)
> + * {
> + *         unsigned long clflush_mask = boot_cpu_data.x86_clflush_size - 1;
> + *         char *vend = (char *)vaddr + size;
> + *         char *p;
> + *
> + *         for (p = (char *)((unsigned long)vaddr & ~clflush_mask);
> + *              p < vend; p += boot_cpu_data.x86_clflush_size)
> + *                 clwb(p);
> + *
> + *         // sfence to order clwb/clflushopt/clflush cache flushes
> + *         // mfence via mb() also works
> + *         wmb();
> + *
> + *         // pcommit and the required sfence for ordering
> + *         pcommit_sfence();
> + * }
> + *
> + * After this function completes the data pointed to by vaddr is has been

s/is //

I fixed it up while applying,

thanks.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: [PATCH] x86: Add kerneldoc for pcommit_sfence()

[Ingo Molnar]

* Borislav Petkov <bp@alien8.de> wrote:

> On Wed, Mar 11, 2015 at 12:19:21PM -0600, Ross Zwisler wrote:
> > Add kerneldoc comments for pcommit_sfence() describing the purpose of
> > the pcommit instruction and demonstrating the usage of that instruction.
> > 
> > Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> > Cc: H Peter Anvin <h.peter.anvin@intel.com>
> > Cc: Ingo Molnar <mingo@kernel.org>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Borislav Petkov <bp@alien8.de>
> > ---
> >  arch/x86/include/asm/special_insns.h | 37 ++++++++++++++++++++++++++++++++++++
> >  1 file changed, 37 insertions(+)
> > 
> > diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
> > index aeb4666e0c0a..1ae81757c05b 100644
> > --- a/arch/x86/include/asm/special_insns.h
> > +++ b/arch/x86/include/asm/special_insns.h
> > @@ -215,6 +215,43 @@ static inline void clwb(volatile void *__p)
> >  		: [pax] "a" (p));
> >  }
> >  
> > +/**
> > + * pcommit_sfence() - persistent commit and fence
> > + *
> > + * The pcommit instruction ensures that data that has been flushed from the
> > + * processor's cache hierarchy with clwb, clflushopt or clflush is accepted to
> > + * memory and is durable on the DIMM.  The primary use case for this is
> > + * persistent memory.

Please capitalize canonical instruction names like the CPU makers do, 
so that they stand out better in free flowing English text, i.e. 
something like:

 *
 * The PCOMMIT instruction ensures that data that has been flushed from the
 * processor's cache hierarchy with CLWB, CLFLUSHOPT or CLFLUSH is accepted to
 * memory and is durable on the DIMM.  The primary use case for this is
 * persistent memory.
 *

> > + * This function shows how to properly use clwb/clflushopt/clflush and pcommit
> > + * with appropriate fencing:

Ditto.

> > + *
> > + * void flush_and_commit_buffer(void *vaddr, unsigned int size)
> > + * {
> > + *         unsigned long clflush_mask = boot_cpu_data.x86_clflush_size - 1;
> > + *         char *vend = (char *)vaddr + size;

So here we cast vaddr to (char *) - which is unnecessary, as 'void *' 
has byte granular pointer arithmetics.

And 'vend' should be void *' to begin with, to match the type 
of 'vaddr'.

> > + *         char *p;

Ditto.

> > + *
> > + *         for (p = (char *)((unsigned long)vaddr & ~clflush_mask);
> > + *              p < vend; p += boot_cpu_data.x86_clflush_size)
> > + *                 clwb(p);
> > + *
> > + *         // sfence to order clwb/clflushopt/clflush cache flushes
> > + *         // mfence via mb() also works

Yeah so this isn't a C++ kernel, thank all the 3000+ gods and other 
supreme beings worshipped on this planet!

> > + *         wmb();
> > + *
> > + *         // pcommit and the required sfence for ordering

Ditto.

> > + *         pcommit_sfence();
> > + * }
> > + *
> > + * After this function completes the data pointed to by vaddr is has been
> 
> s/is //
> 
> I fixed it up while applying,

Also please put 'vaddr' into single quotes, to make the parameter name 
stand out better in written text:

> > + * After this function completes the data pointed to by 'vaddr' has been

Thanks,

	Ingo

Re: [PATCH] x86: Add kerneldoc for pcommit_sfence()

[Ross Zwisler]

On Thu, 2015-03-12 at 11:58 +0100, Ingo Molnar wrote:
> > > +/**
> > > + * pcommit_sfence() - persistent commit and fence
> > > + *
> > > + * The pcommit instruction ensures that data that has been flushed from the
> > > + * processor's cache hierarchy with clwb, clflushopt or clflush is accepted to
> > > + * memory and is durable on the DIMM.  The primary use case for this is
> > > + * persistent memory.
> 
> Please capitalize canonical instruction names like the CPU makers do, 
> so that they stand out better in free flowing English text, i.e. 
> something like:
> 
>  *
>  * The PCOMMIT instruction ensures that data that has been flushed from the
>  * processor's cache hierarchy with CLWB, CLFLUSHOPT or CLFLUSH is accepted to
>  * memory and is durable on the DIMM.  The primary use case for this is
>  * persistent memory.

Sure, will do.

> > > + * void flush_and_commit_buffer(void *vaddr, unsigned int size)
> > > + * {
> > > + *         unsigned long clflush_mask = boot_cpu_data.x86_clflush_size - 1;
> > > + *         char *vend = (char *)vaddr + size;
> 
> So here we cast vaddr to (char *) - which is unnecessary, as 'void *' 
> has byte granular pointer arithmetics.
> 
> And 'vend' should be void *' to begin with, to match the type 
> of 'vaddr'.

The original version, copied in part from clflush_cache_range, did do
everything with void* pointers.  I changed it to use char* pointers based on
feedback from hpa.  :)

It seems like both have arguments for them.  Char pointer arithmetic has the
advantage that its behavior is standard in C, so it's not specific to gcc.  I
agree that void* has the advantage that it fits more naturally with the types
of the parameters passed in, requiring no casting.

I honestly don't feel strongly either way - please let me know what you guys
prefer in the x86 arch code.

> > > + *         for (p = (char *)((unsigned long)vaddr & ~clflush_mask);
> > > + *              p < vend; p += boot_cpu_data.x86_clflush_size)
> > > + *                 clwb(p);
> > > + *
> > > + *         // sfence to order clwb/clflushopt/clflush cache flushes
> > > + *         // mfence via mb() also works
> 
> Yeah so this isn't a C++ kernel, thank all the 3000+ gods and other 
> supreme beings worshipped on this planet!

Yep.  C++ style // comments are happily accepted by gcc in C code, though, and
this was my attempt to get around the fact that /* */ style comments can't be
nested.  I couldn't think of a more elegant way of having code + comments in a
kerneldoc comment.  I agree that if this code were ever to be pulled out and
used, the comment style would need to be corrected to be the standard kernel
style.

> Also please put 'vaddr' into single quotes, to make the parameter name 
> stand out better in written text:
> 
> > > + * After this function completes the data pointed to by 'vaddr' has been

Sure.

Thanks,
- Ross

Re: [PATCH] x86: Add kerneldoc for pcommit_sfence()

[Ingo Molnar]

* Ross Zwisler <ross.zwisler@linux.intel.com> wrote:

> > > > + * void flush_and_commit_buffer(void *vaddr, unsigned int size)
> > > > + * {
> > > > + *         unsigned long clflush_mask = boot_cpu_data.x86_clflush_size - 1;
> > > > + *         char *vend = (char *)vaddr + size;
> > 
> > So here we cast vaddr to (char *) - which is unnecessary, as 'void *' 
> > has byte granular pointer arithmetics.
> > 
> > And 'vend' should be void *' to begin with, to match the type 
> > of 'vaddr'.
> 
> The original version, copied in part from clflush_cache_range, did do
> everything with void* pointers.  I changed it to use char* pointers based on
> feedback from hpa.  :)

:-/

Not sure what hpa's problem with 'void *' was: especially in MM code 
we are using 'void *' rather widely.

All compilers that aim for being able to build the Linux kernel 
implement 'void *' as well, so that 'standard C' argument is pretty 
weak IMHO - unlike some of the more esoteric GCC extensions, this one 
is actually pretty well done and widely used in and outside of the 
kernel.

> It seems like both have arguments for them.  Char pointer arithmetic 
> has the advantage that its behavior is standard in C, so it's not 
> specific to gcc.  I agree that void* has the advantage that it fits 
> more naturally with the types of the parameters passed in, requiring 
> no casting.

It's also a bonus property of 'void *' that unlike 'char *' it cannot 
be dereferenced. So we use it for opaque buffers wherever we can.

> > > > + *         for (p = (char *)((unsigned long)vaddr & ~clflush_mask);
> > > > + *              p < vend; p += boot_cpu_data.x86_clflush_size)
> > > > + *                 clwb(p);
> > > > + *
> > > > + *         // sfence to order clwb/clflushopt/clflush cache flushes
> > > > + *         // mfence via mb() also works
> > 
> > Yeah so this isn't a C++ kernel, thank all the 3000+ gods and other 
> > supreme beings worshipped on this planet!
> 
> Yep.  C++ style // comments are happily accepted by gcc in C code, though, and

GCC accepts other C++ braindamage as well, it doesn't mean we should 
use them. But:

> this was my attempt to get around the fact that /* */ style comments can't be
> nested.  I couldn't think of a more elegant way of having code + comments in a
> kerneldoc comment.  I agree that if this code were ever to be pulled out and
> used, the comment style would need to be corrected to be the standard kernel
> style.

I see, I didn't realize the recursion complication with DocBook - so 
this bit is fine.

Thanks,

	Ingo

Re: [PATCH] x86: Add kerneldoc for pcommit_sfence()

[H. Peter Anvin]

On 03/16/2015 01:35 AM, Ingo Molnar wrote:
> 
> :-/
> 
> Not sure what hpa's problem with 'void *' was: especially in MM code 
> we are using 'void *' rather widely.
> 
> All compilers that aim for being able to build the Linux kernel 
> implement 'void *' as well, so that 'standard C' argument is pretty 
> weak IMHO - unlike some of the more esoteric GCC extensions, this one 
> is actually pretty well done and widely used in and outside of the 
> kernel.
> 
>> It seems like both have arguments for them.  Char pointer arithmetic 
>> has the advantage that its behavior is standard in C, so it's not 
>> specific to gcc.  I agree that void* has the advantage that it fits 
>> more naturally with the types of the parameters passed in, requiring 
>> no casting.
> 
> It's also a bonus property of 'void *' that unlike 'char *' it cannot 
> be dereferenced. So we use it for opaque buffers wherever we can.
> 

The issue isn't void *, it is doing arithmetic on void *.

	-hpa

Re: [PATCH] x86: Add kerneldoc for pcommit_sfence()

[Ingo Molnar]

* H. Peter Anvin <h.peter.anvin@intel.com> wrote:

> On 03/16/2015 01:35 AM, Ingo Molnar wrote:
> > 
> > :-/
> > 
> > Not sure what hpa's problem with 'void *' was: especially in MM code 
> > we are using 'void *' rather widely.
> > 
> > All compilers that aim for being able to build the Linux kernel 
> > implement 'void *' as well, so that 'standard C' argument is 
> > pretty weak IMHO - unlike some of the more esoteric GCC 
> > extensions, this one is actually pretty well done and widely used 
> > in and outside of the kernel.
> > 
> >> It seems like both have arguments for them.  Char pointer 
> >> arithmetic has the advantage that its behavior is standard in C, 
> >> so it's not specific to gcc.  I agree that void* has the 
> >> advantage that it fits more naturally with the types of the 
> >> parameters passed in, requiring no casting.
> > 
> > It's also a bonus property of 'void *' that unlike 'char *' it 
> > cannot be dereferenced. So we use it for opaque buffers wherever 
> > we can.
> 
> The issue isn't void *, it is doing arithmetic on void *.

Mind explaining it to me a bit more verbosely, because I don't think I 
get your point? In my experience arithmetics on void * works just fine 
in the cases I tried.

Thanks,

	Ingo