some question about TCR setting

some question about TCR setting

[vichy]

hi all:
from armv8 spec, no matter TCR_EL1/2/3, there are
1. SH (shareability)
2. ORGN (outer cacheability)
3. IRGN(inner cacheability)

but in the each level block/page descriptor, there are also
sharability, inner, outter, cacheability as well.

What is the difference between the sharability, inner/outer
cacheability between TCR and Block/page descriptors?

Why are there attributes setting keep in TCR?

appreciate your kind help in advance,

some question about TCR setting

[Mark Rutland]

On Fri, Nov 14, 2014 at 07:22:01AM +0000, vichy wrote:
> hi all:
> from armv8 spec, no matter TCR_EL1/2/3, there are
> 1. SH (shareability)
> 2. ORGN (outer cacheability)
> 3. IRGN(inner cacheability)
> 
> but in the each level block/page descriptor, there are also
> sharability, inner, outter, cacheability as well.
> 
> What is the difference between the sharability, inner/outer
> cacheability between TCR and Block/page descriptors?

The {SH,ORGN,IRGN}{1,0} fields in the TCR define which attributes the
CPU will use when walking page tables, while the attributes in the
tables themselves apply to other memory accesses.

> Why are there attributes setting keep in TCR?

The CPU needs to know the attributes to read the page tables with.

The CPU can't read the attributes from the page tables, because in order
to do so it would need to know the attributes to access the page tables
with. Additionally, the page tables might not always be mapped into the
virtual address space.

Mark.

some question about TCR setting

[vichy]

hi Mark:
> The CPU can't read the attributes from the page tables, because in order
> to do so it would need to know the attributes to access the page tables
> with. Additionally, the page tables might not always be mapped into the
> virtual address space.
Per your explanation, if the page table base address in TTBR is
0x1000000, {SH,ORGN,IRGN}{1,0} tell cpu how to access this part of
physical address, right?

if so, I have some questions:
1. The processor use {SH,ORGN,IRGN}{1,0} in TCR to access all the
following page directories?
2. under what circumstance, the page tables might not always be mapped
into the virtual address space.
3. if the {SH,ORGN,IRGN}{1,0} of block/page descriptor is conflict
with that one in TCR. Will that make cache maintenance issue. Or there
are separate cache for page table walking?

Sincerely appreciate your kind help,

some question about TCR setting

[Mark Rutland]

On Fri, Nov 14, 2014 at 01:13:12PM +0000, vichy wrote:
> hi Mark:
> > The CPU can't read the attributes from the page tables, because in order
> > to do so it would need to know the attributes to access the page tables
> > with. Additionally, the page tables might not always be mapped into the
> > virtual address space.
> Per your explanation, if the page table base address in TTBR is
> 0x1000000, {SH,ORGN,IRGN}{1,0} tell cpu how to access this part of
> physical address, right?

If that were TTBR0_ELx {SH,ORGN,IRGN}0 would apply, and for TTBR1_ELx
{SH,ORGN,IRGN}1 would apply. 

> if so, I have some questions:
> 1. The processor use {SH,ORGN,IRGN}{1,0} in TCR to access all the
> following page directories?

Yes.

> 2. under what circumstance, the page tables might not always be mapped
> into the virtual address space.

Consider CONFIG_HIGHPTE. The physical address space can be larger than
the virtual address space, in which case not everything can be
permanently mapped.

> 3. if the {SH,ORGN,IRGN}{1,0} of block/page descriptor is conflict
> with that one in TCR. Will that make cache maintenance issue.

If page tables are accessed with different attributes from those used to
walk them (i.e. those programmed into the TCR), the usual coherency
issues would apply.

Mark.

some question about TCR setting

[vichy]

hi Mark:

> Consider CONFIG_HIGHPTE. The physical address space can be larger than
> the virtual address space, in which case not everything can be
> permanently mapped.
if those page tables are not mapped into the virtual address space,
how OS create them?
(AFAIK, once System.M is enabled, the processor cannot access where
page table doesn't map to)
except those page tables are created before System.M enabled.

>
>> 3. if the {SH,ORGN,IRGN}{1,0} of block/page descriptor is conflict
>> with that one in TCR. Will that make cache maintenance issue.
>
> If page tables are accessed with different attributes from those used to
> walk them (i.e. those programmed into the TCR), the usual coherency
> issues would apply.
BTW, why we need to set page table walk attribute as sharable and
inner/outer cacheable?
Does that mean if core0's page table doesn't contain physical address
0x200000, but core1's page table did and set its attribute as
sharable.
Core0 can access this area without page fault?

Sincerely appreciate your kind help,

some question about TCR setting

[Russell King - ARM Linux]

On Sun, Nov 16, 2014 at 06:53:57PM +0800, vichy wrote:
> hi Mark:
> 
> > Consider CONFIG_HIGHPTE. The physical address space can be larger than
> > the virtual address space, in which case not everything can be
> > permanently mapped.
> if those page tables are not mapped into the virtual address space,
> how OS create them?
> (AFAIK, once System.M is enabled, the processor cannot access where
> page table doesn't map to)
> except those page tables are created before System.M enabled.

The OS temporarily maps the page tables into virtual space to update
them.  Once updated, the OS unmaps the page tables.

However, forget this detail - the MMU page table walker knows nothing
about the virtual address space: MMU page table walks do not happen in
the virtual address space, they happen in the physical address space.
So, it makes no odds what so ever whether the table is mapped or not.

> BTW, why we need to set page table walk attribute as sharable and
> inner/outer cacheable?

The access type is included on the bus along with the address and other
attributes.  This includes whether it's sharable, and the cache attributes.
A MMU page walker may be implemented such that it is capable of accessing
L2 cache.

In that case, we would want the MMU page walker to be able to read from
the L2 cache, which would need the access to be marked as "normal memory,
cacheable" so that the cache is searched for a matching line.  This in
turn means that software does not have to flush page table updates all
the way back to the physical memory - merely flushing them out of the L1
cache is sufficient.

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

some question about TCR setting

[vichy]

hi Russel:
>> if those page tables are not mapped into the virtual address space,
>> how OS create them?
>> (AFAIK, once System.M is enabled, the processor cannot access where
>> page table doesn't map to)
>> except those page tables are created before System.M enabled.
>
> The OS temporarily maps the page tables into virtual space to update
> them.  Once updated, the OS unmaps the page tables.
>
> However, forget this detail - the MMU page table walker knows nothing
> about the virtual address space: MMU page table walks do not happen in
> the virtual address space, they happen in the physical address space.
> So, it makes no odds what so ever whether the table is mapped or not.

For some processors, if the type of L1 cache is VIPT, does that mean
MMU page table will not be cached in this level of cache?

>
>> BTW, why we need to set page table walk attribute as sharable and
>> inner/outer cacheable?
>
> The access type is included on the bus along with the address and other
> attributes.  This includes whether it's sharable, and the cache attributes.
> A MMU page walker may be implemented such that it is capable of accessing
> L2 cache.
>
> In that case, we would want the MMU page walker to be able to read from
> the L2 cache, which would need the access to be marked as "normal memory,
> cacheable" so that the cache is searched for a matching line.  This in
> turn means that software does not have to flush page table updates all
> the way back to the physical memory - merely flushing them out of the L1
> cache is sufficient.
so Inner/outer attributes are use to determine whether it is cached in
L1 or L2 cache, right?
how about sharable?
AFAIK, sharable means for a particular domain it requires hardware to
ensure that
the location is coherent for all agents in that domain.
Since page table is unique to each processor, why we need to share it
between other cores?

Sincerely Appreciate your kind help,

some question about TCR setting

[Mark Rutland]

Hi,

The below applies to ARMv8 (and also ARMv7). Prior versions of the ARM
architecture do not provide the same guarantees.

On Sun, Nov 16, 2014 at 04:52:11PM +0000, Russell King - ARM Linux wrote:
> On Sun, Nov 16, 2014 at 06:53:57PM +0800, vichy wrote:
> > hi Mark:
> > 
> > > Consider CONFIG_HIGHPTE. The physical address space can be larger than
> > > the virtual address space, in which case not everything can be
> > > permanently mapped.
> > if those page tables are not mapped into the virtual address space,
> > how OS create them?
> > (AFAIK, once System.M is enabled, the processor cannot access where
> > page table doesn't map to)
> > except those page tables are created before System.M enabled.
> 
> The OS temporarily maps the page tables into virtual space to update
> them.  Once updated, the OS unmaps the page tables.
> 
> However, forget this detail - the MMU page table walker knows nothing
> about the virtual address space: MMU page table walks do not happen in
> the virtual address space, they happen in the physical address space.
> So, it makes no odds what so ever whether the table is mapped or not.
> 
> > BTW, why we need to set page table walk attribute as sharable and
> > inner/outer cacheable?
> 
> The access type is included on the bus along with the address and other
> attributes.  This includes whether it's sharable, and the cache attributes.
> A MMU page walker may be implemented such that it is capable of accessing
> L2 cache.

Or L1, depending on the implementation of the memory system. The logic
which walks the page tables is essentially another coherent observer, so
it doesn't necessarily matter precisely where it is attached.

> In that case, we would want the MMU page walker to be able to read from
> the L2 cache, which would need the access to be marked as "normal memory,
> cacheable" so that the cache is searched for a matching line.  This in
> turn means that software does not have to flush page table updates all
> the way back to the physical memory - merely flushing them out of the L1
> cache is sufficient.

To further this point, provided the TCR is programmed with the same
attributes the kernel uses to access the page tables, no cache
maintenance is necessary, as the CPU(s) and page table walker(s) are
coherent.

Barriers are necessary to ensure visibility of any page table
modifications prior to explicit memory accesses and/or TLB maintenance.

This is what we do for ARMv7 and arm64.

Thanks,
Mark.

some question about TCR setting

[Russell King - ARM Linux]

On Mon, Nov 17, 2014 at 12:09:52PM +0000, Mark Rutland wrote:
> On Sun, Nov 16, 2014 at 04:52:11PM +0000, Russell King - ARM Linux wrote:
> > The access type is included on the bus along with the address and other
> > attributes.  This includes whether it's sharable, and the cache attributes.
> > A MMU page walker may be implemented such that it is capable of accessing
> > L2 cache.
> 
> Or L1, depending on the implementation of the memory system. The logic
> which walks the page tables is essentially another coherent observer, so
> it doesn't necessarily matter precisely where it is attached.
...
> To further this point, provided the TCR is programmed with the same
> attributes the kernel uses to access the page tables, no cache
> maintenance is necessary, as the CPU(s) and page table walker(s) are
> coherent.

That depends on the implementation - whether the MMU page table walker
can read from the L1 cache.  If you were correct in all cases, we could
get rid of:

        ALT_UP (mcr     p15, 0, r0, c7, c10, 1)         @ flush_pte

in proc-v7-2level.S - but obviously we think that ARMv7 is unable to read
from L1 when walking the page tables.

> This is what we do for ARMv7 and arm64.

>From the code, it looks like we don't for ARMv7.

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

some question about TCR setting

[Mark Rutland]

On Mon, Nov 17, 2014 at 01:03:06PM +0000, Russell King - ARM Linux wrote:
> On Mon, Nov 17, 2014 at 12:09:52PM +0000, Mark Rutland wrote:
> > On Sun, Nov 16, 2014 at 04:52:11PM +0000, Russell King - ARM Linux wrote:
> > > The access type is included on the bus along with the address and other
> > > attributes.  This includes whether it's sharable, and the cache attributes.
> > > A MMU page walker may be implemented such that it is capable of accessing
> > > L2 cache.
> > 
> > Or L1, depending on the implementation of the memory system. The logic
> > which walks the page tables is essentially another coherent observer, so
> > it doesn't necessarily matter precisely where it is attached.
> ...
> > To further this point, provided the TCR is programmed with the same
> > attributes the kernel uses to access the page tables, no cache
> > maintenance is necessary, as the CPU(s) and page table walker(s) are
> > coherent.
> 
> That depends on the implementation - whether the MMU page table walker
> can read from the L1 cache.  If you were correct in all cases, we could
> get rid of:
> 
>         ALT_UP (mcr     p15, 0, r0, c7, c10, 1)         @ flush_pte
> 
> in proc-v7-2level.S - but obviously we think that ARMv7 is unable to read
> from L1 when walking the page tables.
> 
> > This is what we do for ARMv7 and arm64.
> 
> From the code, it looks like we don't for ARMv7.

My bad, I was assuming ARMv7 with the multiprocessor extensions rather
than baseline ARMv7.

Thanks,
Mark,

some question about TCR setting

[Mark Rutland]

On Mon, Nov 17, 2014 at 02:37:18AM +0000, vichy wrote:
> hi Russel:
> >> if those page tables are not mapped into the virtual address space,
> >> how OS create them?
> >> (AFAIK, once System.M is enabled, the processor cannot access where
> >> page table doesn't map to)
> >> except those page tables are created before System.M enabled.
> >
> > The OS temporarily maps the page tables into virtual space to update
> > them.  Once updated, the OS unmaps the page tables.
> >
> > However, forget this detail - the MMU page table walker knows nothing
> > about the virtual address space: MMU page table walks do not happen in
> > the virtual address space, they happen in the physical address space.
> > So, it makes no odds what so ever whether the table is mapped or not.
> 
> For some processors, if the type of L1 cache is VIPT, does that mean
> MMU page table will not be cached in this level of cache?

The page tables could be allocated into any level of cache while they
are mapped (consider speculation by the CPU). Depending on whether the
page table walks are coherent the tables may no be visible to the
walkers from L1.

On ARMv8 (or ARMv7 with multiprocessor extensions) the walks are
coherent, and will look in the L1 if necessary. 

> >> BTW, why we need to set page table walk attribute as sharable and
> >> inner/outer cacheable?
> >
> > The access type is included on the bus along with the address and other
> > attributes.  This includes whether it's sharable, and the cache attributes.
> > A MMU page walker may be implemented such that it is capable of accessing
> > L2 cache.
> >
> > In that case, we would want the MMU page walker to be able to read from
> > the L2 cache, which would need the access to be marked as "normal memory,
> > cacheable" so that the cache is searched for a matching line.  This in
> > turn means that software does not have to flush page table updates all
> > the way back to the physical memory - merely flushing them out of the L1
> > cache is sufficient.
> so Inner/outer attributes are use to determine whether it is cached in
> L1 or L2 cache, right?

Inner and Outer cacheability might not map to L1 and L2. If you are
lucky on ARMv8 CLIDR_EL1.ICB will tell you the boundary between inner
and outer caches, but current implementations are unlikely to.

> how about sharable?
> AFAIK, sharable means for a particular domain it requires hardware to
> ensure that
> the location is coherent for all agents in that domain.
> Since page table is unique to each processor, why we need to share it
> between other cores?

We share page tables between CPUs. On arm64 all CPUs share the same
swappper pgdir in TTBR1_EL1, and if executing threads within the same
task may share the same pgdir in TTBR0_EL1.

Additionally, to prevent issues with mismatched attributes, the TCR
should have the same attributes for walks as we have for mappings of the
memory containing the page tables. We map memory as Normal, Outer
Write-back non-transient, Inner Write-back non-transient, so we must
make page table walks with those same attributes.

Thanks,
Mark.