L3 cache way-locking memory zone

L3 cache way-locking memory zone

[Marek Bykowski]

Hi All,

Would the mainline be interested in definition/creation of the memory
zone to L3 cache way-locking memory range/s?

We have this implemented around the system featuring Cortex-A53/57 and
the CCN-512/504 with L3 cache (LLC) configured to way-locking. The
benefits arising are a faster memory access.

If you reckon this is something of interest I can share more
details, patch, use case, etc.

Thanks,
Marek

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: L3 cache way-locking memory zone

[Sudeep Holla]

On Mon, Apr 15, 2019 at 03:38:49PM +0200, Marek Bykowski wrote:
> Hi All,
>
> Would the mainline be interested in definition/creation of the memory
> zone to L3 cache way-locking memory range/s?
>
> We have this implemented around the system featuring Cortex-A53/57 and
> the CCN-512/504 with L3 cache (LLC) configured to way-locking. The
> benefits arising are a faster memory access.
>
> If you reckon this is something of interest I can share more
> details, patch, use case, etc.
>

I am sure many will be interested. But depends on how generic is the
solution ? It's extremely difficult to come up with one.

ARM has Memory System Resource Partitioning(MPAM) to solve this in a
generic way. I know it may not solve issue for current platforms, but
just thought it's worth pointing out so that people are aware of it.

--
Regards,
Sudeep

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: L3 cache way-locking memory zone

[Marek Bykowski]

Hi Sudeep,

To me it is quite generic but I don't have an experience working on a
broad range of ARM products. 

It works more less along this: one needs to set up an SMC service
allowing querying the L3 way-locking registers/configuration from
within kernel. The L3 way-locking is configured itself statically in
SPL Uboot running EL3, secure state. The static configuration is a
result of the constraint on when and how the registers can be accessed,
namely "normal"/memory access to HN-F cannot be in-flight while the
write to these registers occur - to my best knowledge we cannot meet
that with the system/Linux up and running. The secure state results
from the L3 lock registers being only accessed from the secure. 

After then when Linux boots it does the SMC call to query whether there
is the L3 way-locking in place and what base addresses and the sizes
and it sets up the zone upon it.

The following are some confs/printouts from the Linux having 2G of RAM,
and 24M of LLC with 16M configured to locked:

zone_size[0] 131072 zhole_size[0] 0
zone_size[1] 4096 zhole_size[1] 0
zone_size[2] 389120 zhole_size[2] 0
zone_size[3] 0 zhole_size[3] 0

L3LOCKED zone: 64 pages used for memmap
L3LOCKED zone: 4096 pages, LIFO batch:0

mem_init() DMA->managed_pages 129024
mem_init() L3LOCK->managed_pages 4032
mem_init() Normal->managed_pages 383040
mem_init() Movable->managed_pages 0

And after releasing free pages to the buddy allocator

mem_init() DMA->managed_pages 123787
mem_init() L3LOCK->managed_pages 4096
mem_init() Normal->managed_pages 379305
mem_init() Movable->managed_pages 0

# cat /proc/buddyinfo (reformatted to squeeze)
Node 0, zone DMA    4  4  3  4  2  3  2  4  2  3  122 
Node 0, zone L3LOCK 0  0  0  0  0  0  0  0  0  0  4  
Node 0, zone Normal 21 3  7  2  2  1  1  1  1  2  306 

# cat /proc/slabinfo
[...]
Node 0, zone    L3LOCK
  pages free     4096
        min      135
        low      168
        high     201
   node_scanned  0
        spanned  4096
        present  4096
        managed  4096

Then we can allocate to the L3LOCK zone with 'kmalloc(size,
GFP_L3LOCK)'. 

We have used the L3LOCK to benchmark the DMA that presented the better
results when copying from L3 locked to L3 locked than from "unlocked"
to "unlocked".

Thanks,
Marek

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel