[RFC] MIPS: Add cacheinfo support

[RFC] MIPS: Add cacheinfo support

[justinpopo6]

From: Justin Chen <justin.chen@broadcom.com>

Add cacheinfo support for MIPS architectures.

Use information from the cpuinfo_mips struct to populate the
cacheinfo struct. This allows an architecture agnostic approach,
however this also means if cache information is not properly
populated within the cpuinfo_mips struct, there is nothing
we can do. (I.E. c-r3k.c)

Signed-off-by: Justin Chen <justin.chen@broadcom.com>
---
 arch/mips/kernel/Makefile    |  2 +-
 arch/mips/kernel/cacheinfo.c | 85 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 1 deletion(-)
 create mode 100644 arch/mips/kernel/cacheinfo.c

diff --git a/arch/mips/kernel/Makefile b/arch/mips/kernel/Makefile
index 4a603a3..904a9c4 100644
--- a/arch/mips/kernel/Makefile
+++ b/arch/mips/kernel/Makefile
@@ -7,7 +7,7 @@ extra-y		:= head.o vmlinux.lds
 obj-y		+= cpu-probe.o branch.o elf.o entry.o genex.o idle.o irq.o \
 		   process.o prom.o ptrace.o reset.o setup.o signal.o \
 		   syscall.o time.o topology.o traps.o unaligned.o watch.o \
-		   vdso.o
+		   vdso.o cacheinfo.o
 
 ifdef CONFIG_FUNCTION_TRACER
 CFLAGS_REMOVE_ftrace.o = -pg
diff --git a/arch/mips/kernel/cacheinfo.c b/arch/mips/kernel/cacheinfo.c
new file mode 100644
index 0000000..a92bbba
--- /dev/null
+++ b/arch/mips/kernel/cacheinfo.c
@@ -0,0 +1,85 @@
+/*
+ * MIPS cacheinfo support
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed "as is" WITHOUT ANY WARRANTY of any
+ * kind, whether express or implied; without even the implied warranty
+ * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <linux/cacheinfo.h>
+
+/* Populates leaf and increments to next leaf */
+#define populate_cache(cache, leaf, c_level, c_type)		\
+	leaf->type = c_type;					\
+	leaf->level = c_level;					\
+	leaf->coherency_line_size = c->cache.linesz;		\
+	leaf->number_of_sets = c->cache.sets;			\
+	leaf->ways_of_associativity = c->cache.ways;		\
+	leaf->size = c->cache.linesz * c->cache.sets *		\
+		c->cache.ways;					\
+	leaf++;
+
+static int __init_cache_level(unsigned int cpu)
+{
+	struct cpuinfo_mips *c = &current_cpu_data;
+	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);
+	int levels = 0, leaves = 0;
+
+	/*
+	 * If Dcache is not set, we assume the cache structures
+	 * are not properly initialized.
+	 */
+	if (c->dcache.waysize)
+		levels += 1;
+	else
+		return -ENOENT;
+
+
+	leaves += (c->icache.waysize) ? 2 : 1;
+
+	if (c->scache.waysize) {
+		levels++;
+		leaves++;
+	}
+
+	if (c->tcache.waysize) {
+		levels++;
+		leaves++;
+	}
+
+	this_cpu_ci->num_levels = levels;
+	this_cpu_ci->num_leaves = leaves;
+	return 0;
+}
+
+static int __populate_cache_leaves(unsigned int cpu)
+{
+	struct cpuinfo_mips *c = &current_cpu_data;
+	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);
+	struct cacheinfo *this_leaf = this_cpu_ci->info_list;
+
+	if (c->icache.waysize) {
+		populate_cache(dcache, this_leaf, 1, CACHE_TYPE_DATA);
+		populate_cache(icache, this_leaf, 1, CACHE_TYPE_INST);
+	} else {
+		populate_cache(dcache, this_leaf, 1, CACHE_TYPE_UNIFIED);
+	}
+
+	if (c->scache.waysize)
+		populate_cache(scache, this_leaf, 2, CACHE_TYPE_UNIFIED);
+
+	if (c->tcache.waysize)
+		populate_cache(tcache, this_leaf, 3, CACHE_TYPE_UNIFIED);
+
+	return 0;
+}
+
+DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level)
+DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves)
-- 
2.10.2

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/07/2016 05:16 PM, Justin Chen wrote:
> From: Justin Chen <justin.chen@broadcom.com>
>
> Add cacheinfo support for MIPS architectures.
>
> Use information from the cpuinfo_mips struct to populate the
> cacheinfo struct. This allows an architecture agnostic approach,
> however this also means if cache information is not properly
> populated within the cpuinfo_mips struct, there is nothing
> we can do. (I.E. c-r3k.c)
>
>

Justin, for architecture agnostic approach of work with caches the 
cpuinfo_mips lacks more information:

1.  L1I-L1D coherency status (is L1I coherent to L1D)
2.  L1D-L2 coherency status (is L1D coherent to L2)
3.  L1I-L2 coherency status (is L1I coherent to L2)
4.  L1I refill from L1D (snoops L1D) or L2 (may require L1D flush)?
5.  L1D cache aliasing disabled by HW
6.  L1I cache aliasing disabled by HW
7.  Barrier existence and it's type between various cache flushes as is 
as between cache flush and final (completed) state.
8.  Cache ownership (current: assumed that sibling CPUs share L1 - may 
be shared_cpu_map?)
9.  Is address cache flush operation global in system or pure local?
10. Is index cache flush operation global in system or pure local?

Without that the proposed cpuinfo_mips struct is basically useless.

Regards,
- Leonid

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/07/2016 05:16 PM, Justin Chen wrote:
> From: Justin Chen <justin.chen@broadcom.com>
>
> Add cacheinfo support for MIPS architectures.
>
> Use information from the cpuinfo_mips struct to populate the
> cacheinfo struct. This allows an architecture agnostic approach,
> however this also means if cache information is not properly
> populated within the cpuinfo_mips struct, there is nothing
> we can do. (I.E. c-r3k.c)
>
>

Justin, for architecture agnostic approach of work with caches the 
cpuinfo_mips lacks more information:

1.  L1I-L1D coherency status (is L1I coherent to L1D)
2.  L1D-L2 coherency status (is L1D coherent to L2)
3.  L1I-L2 coherency status (is L1I coherent to L2)
4.  L1I refill from L1D (snoops L1D) or L2 (may require L1D flush)?
5.  L1D cache aliasing disabled by HW
6.  L1I cache aliasing disabled by HW
7.  Barrier existence and it's type between various cache flushes as is 
as between cache flush and final (completed) state.
8.  Cache ownership (current: assumed that sibling CPUs share L1 - may 
be shared_cpu_map?)
9.  Is address cache flush operation global in system or pure local?
10. Is index cache flush operation global in system or pure local?

Without that the proposed cpuinfo_mips struct is basically useless.

Regards,
- Leonid

Re: [RFC] MIPS: Add cacheinfo support

[Justin Chen]

On Thu, Dec 8, 2016 at 3:26 PM, Leonid Yegoshin
<Leonid.Yegoshin@imgtec.com> wrote:
> On 12/07/2016 05:16 PM, Justin Chen wrote:
>>
>> From: Justin Chen <justin.chen@broadcom.com>
>>
>> Add cacheinfo support for MIPS architectures.
>>
>> Use information from the cpuinfo_mips struct to populate the
>> cacheinfo struct. This allows an architecture agnostic approach,
>> however this also means if cache information is not properly
>> populated within the cpuinfo_mips struct, there is nothing
>> we can do. (I.E. c-r3k.c)
>>
>>
>
> Justin, for architecture agnostic approach of work with caches the
> cpuinfo_mips lacks more information:
>
> 1.  L1I-L1D coherency status (is L1I coherent to L1D)
> 2.  L1D-L2 coherency status (is L1D coherent to L2)
> 3.  L1I-L2 coherency status (is L1I coherent to L2)
> 4.  L1I refill from L1D (snoops L1D) or L2 (may require L1D flush)?
> 5.  L1D cache aliasing disabled by HW
> 6.  L1I cache aliasing disabled by HW
> 7.  Barrier existence and it's type between various cache flushes as is as
> between cache flush and final (completed) state.
> 8.  Cache ownership (current: assumed that sibling CPUs share L1 - may be
> shared_cpu_map?)
> 9.  Is address cache flush operation global in system or pure local?
> 10. Is index cache flush operation global in system or pure local?
>
> Without that the proposed cpuinfo_mips struct is basically useless.
>
> Regards,
> - Leonid

Thanks for the comments Leonid.

We should consider the scope of this patch. The information we are
trying to expose to userspace is limited to the struct cacheinfo
located at include/linux/cacheinfo.c (of course this can always be
expanded). So the question is what information would be useful to
expose to userspace.
Some justification for exposing the current information in the
cacheinfo struct could be: (Pulled from another email chain)
"Agreed. So far I have got requests from GCC, JIT and graphics guys.
IIUC they need this to support cache flushing for user applications like
gcc trampolines and JIT compilers. I am also told that having knowledge
of cache architecture can help optimal code strategies, though I don't
have much details on that."
https://patchwork.kernel.org/patch/5867721/

There may be justification for including the points you mentioned
above, but I believe that is outside the scope of this patch. The
cache information exposed in this patch is limited, but I do not
believe it is useless. The points above can be added, but it will
require a rework of the base cacheinfo driver. driver/base/cacheinfo.c

Thanks,
Justin

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/08/2016 04:28 PM, Justin Chen wrote:
> Thanks for the comments Leonid.
>
> We should consider the scope of this patch. The information we are
> trying to expose to userspace is limited to the struct cacheinfo
> located at include/linux/cacheinfo.c (of course this can always be
> expanded). So the question is what information would be useful to
> expose to userspace.
> Some justification for exposing the current information in the
> cacheinfo struct could be: (Pulled from another email chain)
> "Agreed. So far I have got requests from GCC, JIT and graphics guys.
> IIUC they need this to support cache flushing for user applications like
> gcc trampolines and JIT compilers. I am also told that having knowledge
> of cache architecture can help optimal code strategies, though I don't
> have much details on that."
> https://patchwork.kernel.org/patch/5867721/
>
> There may be justification for including the points you mentioned
> above, but I believe that is outside the scope of this patch. The
> cache information exposed in this patch is limited, but I do not
> believe it is useless. The points above can be added, but it will
> require a rework of the base cacheinfo driver. driver/base/cacheinfo.c
>
>
Justin,

CACHE instruction is not available in user space, only SYNCI on MIPS R2+ 
for trampoline.
Any operation with CACHE requires a syscall.

As for SYNCI (trampoline from L1D->L1I) the following information in 
user space is needed:

     1. L1 line size (available via RDHWR $x, $1). It is available now 
directly from CPU, but may be better to supply from kernel because some 
cores has no that.

     2. The flag that L1I is NOT coherent with L1D and SYNCI is needed 
and available

The knowledge about L1/L2 sizes is not needed for regular application... 
well, if application wants to get advantage of cache sizes, well, in 
this case it can be supplied.

But it is unreliable because app may be rescheduled into different kind 
of core which has a different L1 size (in heterogeneous system, BTW). It 
can be fixed by setting affinity, of course (not sure - can it be 
reliably done in BIG/LITTLE approach). But that requires in application 
the knowledge and understanding of system CPU structure... well why we 
can allow all that stuff besides information purpose? It corrupts the 
all efforts and optimization in kernel about performance and powersaving.

Regards,
- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/08/2016 04:28 PM, Justin Chen wrote:
> Thanks for the comments Leonid.
>
> We should consider the scope of this patch. The information we are
> trying to expose to userspace is limited to the struct cacheinfo
> located at include/linux/cacheinfo.c (of course this can always be
> expanded). So the question is what information would be useful to
> expose to userspace.
> Some justification for exposing the current information in the
> cacheinfo struct could be: (Pulled from another email chain)
> "Agreed. So far I have got requests from GCC, JIT and graphics guys.
> IIUC they need this to support cache flushing for user applications like
> gcc trampolines and JIT compilers. I am also told that having knowledge
> of cache architecture can help optimal code strategies, though I don't
> have much details on that."
> https://patchwork.kernel.org/patch/5867721/
>
> There may be justification for including the points you mentioned
> above, but I believe that is outside the scope of this patch. The
> cache information exposed in this patch is limited, but I do not
> believe it is useless. The points above can be added, but it will
> require a rework of the base cacheinfo driver. driver/base/cacheinfo.c
>
>
Justin,

CACHE instruction is not available in user space, only SYNCI on MIPS R2+ 
for trampoline.
Any operation with CACHE requires a syscall.

As for SYNCI (trampoline from L1D->L1I) the following information in 
user space is needed:

     1. L1 line size (available via RDHWR $x, $1). It is available now 
directly from CPU, but may be better to supply from kernel because some 
cores has no that.

     2. The flag that L1I is NOT coherent with L1D and SYNCI is needed 
and available

The knowledge about L1/L2 sizes is not needed for regular application... 
well, if application wants to get advantage of cache sizes, well, in 
this case it can be supplied.

But it is unreliable because app may be rescheduled into different kind 
of core which has a different L1 size (in heterogeneous system, BTW). It 
can be fixed by setting affinity, of course (not sure - can it be 
reliably done in BIG/LITTLE approach). But that requires in application 
the knowledge and understanding of system CPU structure... well why we 
can allow all that stuff besides information purpose? It corrupts the 
all efforts and optimization in kernel about performance and powersaving.

Regards,
- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Florian Fainelli]

On 12/08/2016 05:01 PM, Leonid Yegoshin wrote:
> On 12/08/2016 04:28 PM, Justin Chen wrote:
>> Thanks for the comments Leonid.
>>
>> We should consider the scope of this patch. The information we are
>> trying to expose to userspace is limited to the struct cacheinfo
>> located at include/linux/cacheinfo.c (of course this can always be
>> expanded). So the question is what information would be useful to
>> expose to userspace.
>> Some justification for exposing the current information in the
>> cacheinfo struct could be: (Pulled from another email chain)
>> "Agreed. So far I have got requests from GCC, JIT and graphics guys.
>> IIUC they need this to support cache flushing for user applications like
>> gcc trampolines and JIT compilers. I am also told that having knowledge
>> of cache architecture can help optimal code strategies, though I don't
>> have much details on that."
>> https://patchwork.kernel.org/patch/5867721/
>>
>> There may be justification for including the points you mentioned
>> above, but I believe that is outside the scope of this patch. The
>> cache information exposed in this patch is limited, but I do not
>> believe it is useless. The points above can be added, but it will
>> require a rework of the base cacheinfo driver. driver/base/cacheinfo.c
>>
>>
> Justin,
> 
> CACHE instruction is not available in user space, only SYNCI on MIPS R2+
> for trampoline.
> Any operation with CACHE requires a syscall.
> 
> As for SYNCI (trampoline from L1D->L1I) the following information in
> user space is needed:
> 
>     1. L1 line size (available via RDHWR $x, $1). It is available now
> directly from CPU, but may be better to supply from kernel because some
> cores has no that.
> 
>     2. The flag that L1I is NOT coherent with L1D and SYNCI is needed
> and available

Available but not exposed AFAICT.

> 
> The knowledge about L1/L2 sizes is not needed for regular application...
> well, if application wants to get advantage of cache sizes, well, in
> this case it can be supplied.
> 
> But it is unreliable because app may be rescheduled into different kind
> of core which has a different L1 size (in heterogeneous system, BTW). It
> can be fixed by setting affinity, of course (not sure - can it be
> reliably done in BIG/LITTLE approach). But that requires in application
> the knowledge and understanding of system CPU structure... well why we
> can allow all that stuff besides information purpose? It corrupts the
> all efforts and optimization in kernel about performance and powersaving.

What Justin's patch is about is not so much about providing hints to
user-space to bypass the kernel's own management of caches, (even though
that has been used as an argument by the original introduction of
cacheinfo), but more to provide some information to user-space about the
cache topology and hierarchy.

Even though this is limited information this is still helpful to
applications like lshw and others out there.

What would be needed from your perspective to get cacheinfo added to
MIPS, shall we go back and address your initial comment about all the
little details about coherency, snooping and re-filling strategy?

Thanks
-- 
Florian

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/12/2016 10:24 AM, Florian Fainelli wrote:
>
> What Justin's patch is about is not so much about providing hints to
> user-space to bypass the kernel's own management of caches, (even though
> that has been used as an argument by the original introduction of
> cacheinfo), but more to provide some information to user-space about the
> cache topology and hierarchy.

I missed that, if it is for information purpose only, then it is OK.

>
> Even though this is limited information this is still helpful to
> applications like lshw and others out there.
>
> What would be needed from your perspective to get cacheinfo added to
> MIPS, shall we go back and address your initial comment about all the
> little details about coherency, snooping and re-filling strategy?

It depends. Initially, I thought Justin wants to replace 
arch/mips/mm/c-XXX.c with some universal approach and listed the missed 
stuff for that (I actually missed some more points in that list).

But for information purpose I don't have any more addition to Justin's 
patch... may be the coherency status, it has impact on performance: 
coherency of L1D->L2, L2->memory and L1I->L1D/L2.


- Leonid

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/12/2016 10:24 AM, Florian Fainelli wrote:
>
> What Justin's patch is about is not so much about providing hints to
> user-space to bypass the kernel's own management of caches, (even though
> that has been used as an argument by the original introduction of
> cacheinfo), but more to provide some information to user-space about the
> cache topology and hierarchy.

I missed that, if it is for information purpose only, then it is OK.

>
> Even though this is limited information this is still helpful to
> applications like lshw and others out there.
>
> What would be needed from your perspective to get cacheinfo added to
> MIPS, shall we go back and address your initial comment about all the
> little details about coherency, snooping and re-filling strategy?

It depends. Initially, I thought Justin wants to replace 
arch/mips/mm/c-XXX.c with some universal approach and listed the missed 
stuff for that (I actually missed some more points in that list).

But for information purpose I don't have any more addition to Justin's 
patch... may be the coherency status, it has impact on performance: 
coherency of L1D->L2, L2->memory and L1I->L1D/L2.


- Leonid

Re: [RFC] MIPS: Add cacheinfo support

[Florian Fainelli]

On 12/12/2016 12:45 PM, Leonid Yegoshin wrote:
> On 12/12/2016 10:24 AM, Florian Fainelli wrote:
>>
>> What Justin's patch is about is not so much about providing hints to
>> user-space to bypass the kernel's own management of caches, (even though
>> that has been used as an argument by the original introduction of
>> cacheinfo), but more to provide some information to user-space about the
>> cache topology and hierarchy.
> 
> I missed that, if it is for information purpose only, then it is OK.
> 
>>
>> Even though this is limited information this is still helpful to
>> applications like lshw and others out there.
>>
>> What would be needed from your perspective to get cacheinfo added to
>> MIPS, shall we go back and address your initial comment about all the
>> little details about coherency, snooping and re-filling strategy?
> 
> It depends. Initially, I thought Justin wants to replace
> arch/mips/mm/c-XXX.c with some universal approach and listed the missed
> stuff for that (I actually missed some more points in that list).
> 
> But for information purpose I don't have any more addition to Justin's
> patch... may be the coherency status, it has impact on performance:
> coherency of L1D->L2, L2->memory and L1I->L1D/L2.

OK, how would you want that to be represented? Should we try to "link"
with the leaf we are coherent with? For instance, if the L1D cache is
coherent with the L2, we have something like this:

# Assuming this is L1D cache:
/sys/devices/system/cpu/cpu0/cache/index0
ls -1
coherency_line_size
level
number_of_sets
physical_line_partition
power/
shared_cpu_list
shared_cpu_map
size
type
uevent
ways_of_associativity

We add a new symbolic link, e.g:

coherent_with -> ../index1

that indicates that this cache is coherent with the cache pointed at by
directory index1.

Thanks!
-- 
Florian

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/12/2016 01:57 PM, Florian Fainelli wrote:
> OK, how would you want that to be represented? Should we try to "link"
> with the leaf we are coherent with? For instance, if the L1D cache is
> coherent with the L2, we have something like this:
>
> # Assuming this is L1D cache:
> /sys/devices/system/cpu/cpu0/cache/index0
> ls -1
> coherency_line_size
> level
> number_of_sets
> physical_line_partition
> power/
> shared_cpu_list
> shared_cpu_map
> size
> type
> uevent
> ways_of_associativity
>
> We add a new symbolic link, e.g:
>
> coherent_with -> ../index1
>
> that indicates that this cache is coherent with the cache pointed at by
> directory index1.
>
> Thanks!
Well, I prefer the collection of 3 flags:

flush_required        - or flush to next level is required to force 
coherency between this level and next after update of this level
invalidate_required   - or invalidate is required to force coherency 
between this level and next after update of next level
coherent_to_level     - object is coherent with this level (only one 
domain of coherency on this level is assumed)

So, the standard MIPS L1D could be "flush_required", 
"invalidate_required" and "coherent_to_level", L1I could be 
"invalidate_required" and a fully coherent MIPS I6400 L1D could be just 
"coherent_to_level" (no flush/invalidate required).

The reason why I don't like symlink here because it is more precise 
information and it immediately pulls the question "do we need to have 
horizontal links like CPU0 L1D is coherent with CPU1 L1D?" or "do we 
need to have a horizontal link like CPU0 L1D is coherent with CPU1 L1I?" 
or "do we have a MEMORY object?" or "what if coherency is not 
bidirectional (L1D snoops L2 but not vise versa). And you should 
maintain that links in case of CPU goes on/offline. And pulling info 
from that link net is nontrivial in user level, especially if "lshw" 
works in parallel with CPUx going offline.

But what you need is actually the answer on that 3 questions (see flags 
above). At least I don't see a reason for more complicated cache level 
relationship. Even the case BIG-LITTLE still is covered by flags.

- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/12/2016 01:57 PM, Florian Fainelli wrote:
> OK, how would you want that to be represented? Should we try to "link"
> with the leaf we are coherent with? For instance, if the L1D cache is
> coherent with the L2, we have something like this:
>
> # Assuming this is L1D cache:
> /sys/devices/system/cpu/cpu0/cache/index0
> ls -1
> coherency_line_size
> level
> number_of_sets
> physical_line_partition
> power/
> shared_cpu_list
> shared_cpu_map
> size
> type
> uevent
> ways_of_associativity
>
> We add a new symbolic link, e.g:
>
> coherent_with -> ../index1
>
> that indicates that this cache is coherent with the cache pointed at by
> directory index1.
>
> Thanks!
Well, I prefer the collection of 3 flags:

flush_required        - or flush to next level is required to force 
coherency between this level and next after update of this level
invalidate_required   - or invalidate is required to force coherency 
between this level and next after update of next level
coherent_to_level     - object is coherent with this level (only one 
domain of coherency on this level is assumed)

So, the standard MIPS L1D could be "flush_required", 
"invalidate_required" and "coherent_to_level", L1I could be 
"invalidate_required" and a fully coherent MIPS I6400 L1D could be just 
"coherent_to_level" (no flush/invalidate required).

The reason why I don't like symlink here because it is more precise 
information and it immediately pulls the question "do we need to have 
horizontal links like CPU0 L1D is coherent with CPU1 L1D?" or "do we 
need to have a horizontal link like CPU0 L1D is coherent with CPU1 L1I?" 
or "do we have a MEMORY object?" or "what if coherency is not 
bidirectional (L1D snoops L2 but not vise versa). And you should 
maintain that links in case of CPU goes on/offline. And pulling info 
from that link net is nontrivial in user level, especially if "lshw" 
works in parallel with CPUx going offline.

But what you need is actually the answer on that 3 questions (see flags 
above). At least I don't see a reason for more complicated cache level 
relationship. Even the case BIG-LITTLE still is covered by flags.

- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Maciej W. Rozycki]

On Tue, 13 Dec 2016, Leonid Yegoshin wrote:

> Well, I prefer the collection of 3 flags:
> 
> flush_required        - or flush to next level is required to force coherency
> between this level and next after update of this level
> invalidate_required   - or invalidate is required to force coherency between
> this level and next after update of next level
> coherent_to_level     - object is coherent with this level (only one domain of
> coherency on this level is assumed)

 A "flush" has always been an ambiguous term -- meaning anything from 
"write-back", through "write-back+invalidate" to "invalidate".  What is 
"flush" is supposed to exactly mean here?  If specifically any of the two 
formers (the latter is obviously the other flag), then I suggest using the 
respective name, or otherwise if it is meant to be generic "do what is 
needed", then perhaps "synchronize" will work, like with the name of the 
SYNCI instruction?

  Maciej

Re: [RFC] MIPS: Add cacheinfo support

[Maciej W. Rozycki]

On Tue, 13 Dec 2016, Leonid Yegoshin wrote:

> Well, I prefer the collection of 3 flags:
> 
> flush_required        - or flush to next level is required to force coherency
> between this level and next after update of this level
> invalidate_required   - or invalidate is required to force coherency between
> this level and next after update of next level
> coherent_to_level     - object is coherent with this level (only one domain of
> coherency on this level is assumed)

 A "flush" has always been an ambiguous term -- meaning anything from 
"write-back", through "write-back+invalidate" to "invalidate".  What is 
"flush" is supposed to exactly mean here?  If specifically any of the two 
formers (the latter is obviously the other flag), then I suggest using the 
respective name, or otherwise if it is meant to be generic "do what is 
needed", then perhaps "synchronize" will work, like with the name of the 
SYNCI instruction?

  Maciej

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/13/2016 03:09 AM, Maciej W. Rozycki wrote:
> On Tue, 13 Dec 2016, Leonid Yegoshin wrote:
>
>> Well, I prefer the collection of 3 flags:
>>
>> flush_required        - or flush to next level is required to force coherency
>> between this level and next after update of this level
>> invalidate_required   - or invalidate is required to force coherency between
>> this level and next after update of next level
>> coherent_to_level     - object is coherent with this level (only one domain of
>> coherency on this level is assumed)
>   A "flush" has always been an ambiguous term -- meaning anything from
> "write-back", through "write-back+invalidate" to "invalidate".  What is
> "flush" is supposed to exactly mean here?  If specifically any of the two
> formers (the latter is obviously the other flag), then I suggest using the
> respective name, or otherwise if it is meant to be generic "do what is
> needed", then perhaps "synchronize" will work, like with the name of the
> SYNCI instruction?
>
Well, I am not expert in choosing names, you know. I just would like to 
make a distinguishing between "transfer data from this level to next 
one" - any way - writeback or write-back-invalidate, and "clear data in 
this level to be pulled on demand from next level".

This is an information purpose as Florian described, so details can be 
skipped. The only useful info here - some efforts are needed after event 
(see above) and that efforts are not cheap from performance point of view.

If we put here details then it is a long way because we need a lot of 
details, see my original post. However, I would like to differ "flush" 
and "invalidate" because it is triggered by different kind of event 
(from upstream or from downstream).

- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/13/2016 03:09 AM, Maciej W. Rozycki wrote:
> On Tue, 13 Dec 2016, Leonid Yegoshin wrote:
>
>> Well, I prefer the collection of 3 flags:
>>
>> flush_required        - or flush to next level is required to force coherency
>> between this level and next after update of this level
>> invalidate_required   - or invalidate is required to force coherency between
>> this level and next after update of next level
>> coherent_to_level     - object is coherent with this level (only one domain of
>> coherency on this level is assumed)
>   A "flush" has always been an ambiguous term -- meaning anything from
> "write-back", through "write-back+invalidate" to "invalidate".  What is
> "flush" is supposed to exactly mean here?  If specifically any of the two
> formers (the latter is obviously the other flag), then I suggest using the
> respective name, or otherwise if it is meant to be generic "do what is
> needed", then perhaps "synchronize" will work, like with the name of the
> SYNCI instruction?
>
Well, I am not expert in choosing names, you know. I just would like to 
make a distinguishing between "transfer data from this level to next 
one" - any way - writeback or write-back-invalidate, and "clear data in 
this level to be pulled on demand from next level".

This is an information purpose as Florian described, so details can be 
skipped. The only useful info here - some efforts are needed after event 
(see above) and that efforts are not cheap from performance point of view.

If we put here details then it is a long way because we need a lot of 
details, see my original post. However, I would like to differ "flush" 
and "invalidate" because it is triggered by different kind of event 
(from upstream or from downstream).

- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Justin Chen]

On Tue, Dec 13, 2016 at 11:18 AM, Leonid Yegoshin
<Leonid.Yegoshin@imgtec.com> wrote:
> On 12/13/2016 03:09 AM, Maciej W. Rozycki wrote:
>>
>> On Tue, 13 Dec 2016, Leonid Yegoshin wrote:
>>
>>> Well, I prefer the collection of 3 flags:
>>>
>>> flush_required        - or flush to next level is required to force
>>> coherency
>>> between this level and next after update of this level
>>> invalidate_required   - or invalidate is required to force coherency
>>> between
>>> this level and next after update of next level
>>> coherent_to_level     - object is coherent with this level (only one
>>> domain of
>>> coherency on this level is assumed)
>>
>>   A "flush" has always been an ambiguous term -- meaning anything from
>> "write-back", through "write-back+invalidate" to "invalidate".  What is
>> "flush" is supposed to exactly mean here?  If specifically any of the two
>> formers (the latter is obviously the other flag), then I suggest using the
>> respective name, or otherwise if it is meant to be generic "do what is
>> needed", then perhaps "synchronize" will work, like with the name of the
>> SYNCI instruction?
>>
> Well, I am not expert in choosing names, you know. I just would like to make
> a distinguishing between "transfer data from this level to next one" - any
> way - writeback or write-back-invalidate, and "clear data in this level to
> be pulled on demand from next level".
>
> This is an information purpose as Florian described, so details can be
> skipped. The only useful info here - some efforts are needed after event
> (see above) and that efforts are not cheap from performance point of view.
>
> If we put here details then it is a long way because we need a lot of
> details, see my original post. However, I would like to differ "flush" and
> "invalidate" because it is triggered by different kind of event (from
> upstream or from downstream).
>
> - Leonid.
I can add in the flags mentioned by Leonid. However, I would suggest
these flags be added in as a separate patch series as it will require
modifications to the cacheinfo code to propagate the information to
the sysfs.

Would it be ok to accept this patch series as is while I work with the
cacheinfo maintainers to include these new flags?

-Justin

Re: [RFC] MIPS: Add cacheinfo support

[Florian Fainelli]

On 12/13/2016 12:01 PM, Justin Chen wrote:
> On Tue, Dec 13, 2016 at 11:18 AM, Leonid Yegoshin
> <Leonid.Yegoshin@imgtec.com> wrote:
>> On 12/13/2016 03:09 AM, Maciej W. Rozycki wrote:
>>>
>>> On Tue, 13 Dec 2016, Leonid Yegoshin wrote:
>>>
>>>> Well, I prefer the collection of 3 flags:
>>>>
>>>> flush_required        - or flush to next level is required to force
>>>> coherency
>>>> between this level and next after update of this level
>>>> invalidate_required   - or invalidate is required to force coherency
>>>> between
>>>> this level and next after update of next level
>>>> coherent_to_level     - object is coherent with this level (only one
>>>> domain of
>>>> coherency on this level is assumed)
>>>
>>>   A "flush" has always been an ambiguous term -- meaning anything from
>>> "write-back", through "write-back+invalidate" to "invalidate".  What is
>>> "flush" is supposed to exactly mean here?  If specifically any of the two
>>> formers (the latter is obviously the other flag), then I suggest using the
>>> respective name, or otherwise if it is meant to be generic "do what is
>>> needed", then perhaps "synchronize" will work, like with the name of the
>>> SYNCI instruction?
>>>
>> Well, I am not expert in choosing names, you know. I just would like to make
>> a distinguishing between "transfer data from this level to next one" - any
>> way - writeback or write-back-invalidate, and "clear data in this level to
>> be pulled on demand from next level".
>>
>> This is an information purpose as Florian described, so details can be
>> skipped. The only useful info here - some efforts are needed after event
>> (see above) and that efforts are not cheap from performance point of view.
>>
>> If we put here details then it is a long way because we need a lot of
>> details, see my original post. However, I would like to differ "flush" and
>> "invalidate" because it is triggered by different kind of event (from
>> upstream or from downstream).
>>
>> - Leonid.
> I can add in the flags mentioned by Leonid. However, I would suggest
> these flags be added in as a separate patch series as it will require
> modifications to the cacheinfo code to propagate the information to
> the sysfs.
> 
> Would it be ok to accept this patch series as is while I work with the
> cacheinfo maintainers to include these new flags?

Seems entirely reasonable to me. You may want to make a new patch
submission without "RFC", just to make it clear that this is a patch
ready for review/acceptance.

Thanks!
-- 
Florian

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/13/2016 12:05 PM, Florian Fainelli wrote:
> On 12/13/2016 12:01 PM, Justin Chen wrote:
>> On Tue, Dec 13, 2016 at 11:18 AM, Leonid Yegoshin
>> <Leonid.Yegoshin@imgtec.com> wrote:
>>> On 12/13/2016 03:09 AM, Maciej W. Rozycki wrote:
>>>> On Tue, 13 Dec 2016, Leonid Yegoshin wrote:
>>>>
>>>>> Well, I prefer the collection of 3 flags:
>>>>>
>>>>> flush_required        - or flush to next level is required to force
>>>>> coherency
>>>>> between this level and next after update of this level
>>>>> invalidate_required   - or invalidate is required to force coherency
>>>>> between
>>>>> this level and next after update of next level
>>>>> coherent_to_level     - object is coherent with this level (only one
>>>>> domain of
>>>>> coherency on this level is assumed)
>>>>    A "flush" has always been an ambiguous term -- meaning anything from
>>>> "write-back", through "write-back+invalidate" to "invalidate".  What is
>>>> "flush" is supposed to exactly mean here?  If specifically any of the two
>>>> formers (the latter is obviously the other flag), then I suggest using the
>>>> respective name, or otherwise if it is meant to be generic "do what is
>>>> needed", then perhaps "synchronize" will work, like with the name of the
>>>> SYNCI instruction?
>>>>
>>> Well, I am not expert in choosing names, you know. I just would like to make
>>> a distinguishing between "transfer data from this level to next one" - any
>>> way - writeback or write-back-invalidate, and "clear data in this level to
>>> be pulled on demand from next level".
>>>
>>> This is an information purpose as Florian described, so details can be
>>> skipped. The only useful info here - some efforts are needed after event
>>> (see above) and that efforts are not cheap from performance point of view.
>>>
>>> If we put here details then it is a long way because we need a lot of
>>> details, see my original post. However, I would like to differ "flush" and
>>> "invalidate" because it is triggered by different kind of event (from
>>> upstream or from downstream).
>>>
>>> - Leonid.
>> I can add in the flags mentioned by Leonid. However, I would suggest
>> these flags be added in as a separate patch series as it will require
>> modifications to the cacheinfo code to propagate the information to
>> the sysfs.
>>
>> Would it be ok to accept this patch series as is while I work with the
>> cacheinfo maintainers to include these new flags?
> Seems entirely reasonable to me. You may want to make a new patch
> submission without "RFC", just to make it clear that this is a patch
> ready for review/acceptance.
>
> Thanks!
I agree.

- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Leonid Yegoshin]

On 12/13/2016 12:05 PM, Florian Fainelli wrote:
> On 12/13/2016 12:01 PM, Justin Chen wrote:
>> On Tue, Dec 13, 2016 at 11:18 AM, Leonid Yegoshin
>> <Leonid.Yegoshin@imgtec.com> wrote:
>>> On 12/13/2016 03:09 AM, Maciej W. Rozycki wrote:
>>>> On Tue, 13 Dec 2016, Leonid Yegoshin wrote:
>>>>
>>>>> Well, I prefer the collection of 3 flags:
>>>>>
>>>>> flush_required        - or flush to next level is required to force
>>>>> coherency
>>>>> between this level and next after update of this level
>>>>> invalidate_required   - or invalidate is required to force coherency
>>>>> between
>>>>> this level and next after update of next level
>>>>> coherent_to_level     - object is coherent with this level (only one
>>>>> domain of
>>>>> coherency on this level is assumed)
>>>>    A "flush" has always been an ambiguous term -- meaning anything from
>>>> "write-back", through "write-back+invalidate" to "invalidate".  What is
>>>> "flush" is supposed to exactly mean here?  If specifically any of the two
>>>> formers (the latter is obviously the other flag), then I suggest using the
>>>> respective name, or otherwise if it is meant to be generic "do what is
>>>> needed", then perhaps "synchronize" will work, like with the name of the
>>>> SYNCI instruction?
>>>>
>>> Well, I am not expert in choosing names, you know. I just would like to make
>>> a distinguishing between "transfer data from this level to next one" - any
>>> way - writeback or write-back-invalidate, and "clear data in this level to
>>> be pulled on demand from next level".
>>>
>>> This is an information purpose as Florian described, so details can be
>>> skipped. The only useful info here - some efforts are needed after event
>>> (see above) and that efforts are not cheap from performance point of view.
>>>
>>> If we put here details then it is a long way because we need a lot of
>>> details, see my original post. However, I would like to differ "flush" and
>>> "invalidate" because it is triggered by different kind of event (from
>>> upstream or from downstream).
>>>
>>> - Leonid.
>> I can add in the flags mentioned by Leonid. However, I would suggest
>> these flags be added in as a separate patch series as it will require
>> modifications to the cacheinfo code to propagate the information to
>> the sysfs.
>>
>> Would it be ok to accept this patch series as is while I work with the
>> cacheinfo maintainers to include these new flags?
> Seems entirely reasonable to me. You may want to make a new patch
> submission without "RFC", just to make it clear that this is a patch
> ready for review/acceptance.
>
> Thanks!
I agree.

- Leonid.

Re: [RFC] MIPS: Add cacheinfo support

[Ralf Baechle]

On Mon, Dec 12, 2016 at 12:45:37PM -0800, Leonid Yegoshin wrote:

> On 12/12/2016 10:24 AM, Florian Fainelli wrote:
> > 
> > What Justin's patch is about is not so much about providing hints to
> > user-space to bypass the kernel's own management of caches, (even though
> > that has been used as an argument by the original introduction of
> > cacheinfo), but more to provide some information to user-space about the
> > cache topology and hierarchy.
> 
> I missed that, if it is for information purpose only, then it is OK.

Cache information can also be used for some software optimizations though
I think GCC at this time doesn't do this kind of trickery.

> > Even though this is limited information this is still helpful to
> > applications like lshw and others out there.
> > 
> > What would be needed from your perspective to get cacheinfo added to
> > MIPS, shall we go back and address your initial comment about all the
> > little details about coherency, snooping and re-filling strategy?
> 
> It depends. Initially, I thought Justin wants to replace
> arch/mips/mm/c-XXX.c with some universal approach and listed the missed
> stuff for that (I actually missed some more points in that list).
> 
> But for information purpose I don't have any more addition to Justin's
> patch... may be the coherency status, it has impact on performance:
> coherency of L1D->L2, L2->memory and L1I->L1D/L2.

Coherency is hard to properly describe, there are many shades of grey
between fully coherent and not coherent at all.

If we expose something to userspace which constitutes some kind of API
then we really need to get it right because we can't change it arbitrarily
later on.

  Ralf

Re: [RFC] MIPS: Add cacheinfo support

[Ralf Baechle]

On Thu, Dec 08, 2016 at 05:01:53PM -0800, Leonid Yegoshin wrote:

> CACHE instruction is not available in user space, only SYNCI on MIPS R2+ for
> trampoline.
> Any operation with CACHE requires a syscall.

Even worse, some older CPUs allow the execution of certain CACHE operations
from user space.  This is a feature which can be disabled by kernel.

> As for SYNCI (trampoline from L1D->L1I) the following information in user
> space is needed:
> 
>     1. L1 line size (available via RDHWR $x, $1). It is available now
> directly from CPU, but may be better to supply from kernel because some
> cores has no that.
> 
>     2. The flag that L1I is NOT coherent with L1D and SYNCI is needed and
> available
> 
> The knowledge about L1/L2 sizes is not needed for regular application...
> well, if application wants to get advantage of cache sizes, well, in this
> case it can be supplied.
> 
> But it is unreliable because app may be rescheduled into different kind of
> core which has a different L1 size (in heterogeneous system, BTW). It can be
> fixed by setting affinity, of course (not sure - can it be reliably done in
> BIG/LITTLE approach). But that requires in application the knowledge and
> understanding of system CPU structure... well why we can allow all that
> stuff besides information purpose? It corrupts the all efforts and
> optimization in kernel about performance and powersaving.

Also let's not forget about CPUs like Octeons which have CPUs that don't
quite fit in the usual scheme of doing things.

  Ralf