Some Large Address Space Ponders on A9

Some Large Address Space Ponders on A9

[Jon Loeliger]

Folks,

I have a classic Cortex A9 based SoC in which I will need to
do some device driver development that will be capable of
addressing a physical address space larger than 32-bits.
I understand that the A9 doesn't support LPAE and that
pretending that it does and config'ing LPAE "on" will just
break six-ways-to-hell.

But I need to be able to specify some 64-bit addresses in
the Device Tree, and allow my device driver to manipulate
64-bit resource_size_t ranges.

Here's the problem.  Over in include/linux/types.h we find:

    #ifdef CONFIG_PHYS_ADDR_T_64BIT
    typedef u64 phys_addr_t;
    #else
    typedef u32 phys_addr_t;
    #endif

    typedef phys_addr_t resource_size_t;

So that means on my A9 system, phys_addr_t and resource_size_t
are either both 32-bit or both 64-bit.  I can't get just resource_size_t
to be 64-bit without some surgery here.

I've tried the obvious experiment of just config'ing up a
selection defining CONFIG_PHYS_ADDR_T_64BIT and seeing
if things work.  They don't; details below.

So, a few questions...

Should using a 64-bit phys_addr_t on an A9 "just work"?  Or is
this a new scenario never considered before?  Maybe a bug
due to this odd configuration that might need to be fixed?
Maybe some assumption of alignment, packed-ness
or sizes in some structure relating to context switching?

I think the device-tree handling code is able to grok and process
64-bit addresses.  Does it make sense to allow the phys_addr_t and
the resource_size_t to take on different sizes?  That is, to break the
above definition of resource_size_t from types.h and allow its size
to be determined independently of phys_addr_t?

Is there some easier (obvious even?) way to allow some DT devices to
use 64-bit resources and I've just missed it?

Thanks,
jdl


Failure mode details:

During boot, cpu_v7_switch_mm() gets handed an r1 with 0 in it
and dereferences it.  Here, in proc-v7-2level.S:

/*
 *    cpu_v7_switch_mm(pgd_phys, tsk)
 *
 *    Set the translation table base pointer to be pgd_phys
 *
 *    - pgd_phys - physical address of new TTB
 *
 *    It is assumed that:
 *    - we are not using split page tables
 */
ENTRY(cpu_v7_switch_mm)
#ifdef CONFIG_MMU
    mov    r2, #0
    mmid    r1, r1                @ get mm->context.id
    ALT_SMP(orr    r0, r0, #TTB_FLAGS_SMP)
    ALT_UP(orr    r0, r0, #TTB_FLAGS_UP)
#ifdef CONFIG_ARM_ERRATA_430973
    mcr    p15, 0, r2, c7, c5, 6        @ flush BTAC/BTB
#endif
#ifdef CONFIG_PID_IN_CONTEXTIDR
    mrc    p15, 0, r2, c13, c0, 1        @ read current context ID
    lsr    r2, r2, #8            @ extract the PID
    bfi    r1, r2, #8, #24            @ insert into new context ID
#endif
#ifdef CONFIG_ARM_ERRATA_754322
    dsb
#endif
    mcr    p15, 0, r1, c13, c0, 1        @ set context ID
    isb
    mcr    p15, 0, r0, c2, c0, 0        @ set TTB 0
    isb
#endif
    mov    pc, lr
ENDPROC(cpu_v7_switch_mm)



[    2.724891] devtmpfs: mounted
[    2.728262] Freeing unused kernel memory: 152K (c03df000 - c0405000)
[    2.736104] Unable to handle kernel NULL pointer dereference at
virtual address 00000178
[    2.744293] pgd = ede00000
[    2.747010] [00000178] *pgd=00000000
[    2.750637] Internal error: Oops: 5 [#1] SMP ARM
[    2.755259] Modules linked in:
[    2.758338] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
3.14.0-00019-g5f79c4b-dirty #380
[    2.766264] task: ee84c000 ti: ee858000 task.ti: ee858000
[    2.771679] PC is at cpu_v7_switch_mm+0x4/0x24
[    2.776139] LR is at flush_old_exec+0x3f0/0x514
[    2.780682] pc : [<c0018f64>]    lr : [<c00a8d5c>]    psr: 20000113
[    2.780682] sp : ee859ea8  ip : ede00000  fp : c0429698
[    2.792167] r10: ee84c314  r9 : eea4c800  r8 : eddf4000
[    2.797399] r7 : eeb78400  r6 : 00000000  r5 : ee84c000  r4 : c0415840
[    2.803932] r3 : 00000001  r2 : 00000000  r1 : 00000000  r0 : 2de00000
[    2.810466] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM
Segment kernel
[    2.817782] Control: 10c5387d  Table: 0000404a  DAC: 00000015
[    2.823534] Process swapper/0 (pid: 1, stack limit = 0xee858240)
[    2.829547] Stack: (0xee859ea8 to 0xee85a000)
[    2.833917] 9ea0:                   05000002 ee8b96b4 ee8b9680
eeb78400 00000001 00000002
[    2.842109] 9ec0: ee004100 c00e1950 eeb78400 00000080 00000001
ee858000 00000000 eeb78400
[    2.850301] 9ee0: 00000001 beffffd3 ee004f00 eeff0075 eeb78400
c00a7d3c 00000017 ee859f0c
[    2.858491] 9f00: 00000000 0000000b befff000 ee859ec8 beffffc8
c0400fc8 0000000b 00000000
[    2.866683] 9f20: ef003000 eeb78400 c04177a4 c00e1698 00000001
c0417168 eddf4044 eddf4000
[    2.874874] 9f40: c0429698 c00a7e9c eeb78400 ee858000 ee81c000
00000000 00000001 c00a9498
[    2.883065] 9f60: c0410258 c04102e0 00000000 ee84c1fc 00000000
c04291c0 c0364a08 00000000
[    2.891255] 9f80: 00000000 00000000 00000000 00000000 00000000
c00086d4 00000000 c04291c0
[    2.899444] 9fa0: c02cb744 c02cb7bc 00000000 c000d918 00000000
00000000 00000000 00000000
[    2.907633] 9fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[    2.915822] 9fe0: 00000000 00000000 00000000 00000000 00000013
00000000 ffffffff ffffffff
[    2.924029] [<c0018f64>] (cpu_v7_switch_mm) from [<ee858000>] (0xee858000)
[    2.930920] Code: 00000000 00000000 00000000 e3a02000 (e5911178)
[    2.937053] ---[ end trace 41f2c02886b01a23 ]---

Some Large Address Space Ponders on A9

[Catalin Marinas]

On Tue, Jul 01, 2014 at 10:59:45AM -0500, Jon Loeliger wrote:
> I have a classic Cortex A9 based SoC in which I will need to
> do some device driver development that will be capable of
> addressing a physical address space larger than 32-bits.
> I understand that the A9 doesn't support LPAE and that
> pretending that it does and config'ing LPAE "on" will just
> break six-ways-to-hell.
> 
> But I need to be able to specify some 64-bit addresses in
> the Device Tree, and allow my device driver to manipulate
> 64-bit resource_size_t ranges.
> 
> Here's the problem.  Over in include/linux/types.h we find:
> 
>     #ifdef CONFIG_PHYS_ADDR_T_64BIT
>     typedef u64 phys_addr_t;
>     #else
>     typedef u32 phys_addr_t;
>     #endif
> 
>     typedef phys_addr_t resource_size_t;

I won't comment on how you define this in DT but for the kernel things
will likely break when you increase phys_addr_t to 64-bit on non-LPAE
builds.

First (only) question - how do you intend to map such physical address?
The only way I'm aware of on A9 is using supersections and the ARM Linux
port provides __arm_ioremap_pfn(). However, the supersections code is
only enabled if !SMP.

-- 
Catalin

Some Large Address Space Ponders on A9

[Arnd Bergmann]

On Tuesday 01 July 2014 10:59:45 Jon Loeliger wrote:
> Folks,
> 
> I have a classic Cortex A9 based SoC in which I will need to
> do some device driver development that will be capable of
> addressing a physical address space larger than 32-bits.
> I understand that the A9 doesn't support LPAE and that
> pretending that it does and config'ing LPAE "on" will just
> break six-ways-to-hell.
> 
> But I need to be able to specify some 64-bit addresses in
> the Device Tree, and allow my device driver to manipulate
> 64-bit resource_size_t ranges.
> 
> Here's the problem.  Over in include/linux/types.h we find:
> 
>     #ifdef CONFIG_PHYS_ADDR_T_64BIT
>     typedef u64 phys_addr_t;
>     #else
>     typedef u32 phys_addr_t;
>     #endif
> 
>     typedef phys_addr_t resource_size_t;
> 
> So that means on my A9 system, phys_addr_t and resource_size_t
> are either both 32-bit or both 64-bit.  I can't get just resource_size_t
> to be 64-bit without some surgery here.

Hmm, we sometimes confuse phys_addr_t and dma_addr_t, because for
all practical purposes they tend to be the same. However, I think
what you need here is dma_addr_t, which is defined independently.

You will probably run into bugs if you have dma_addr_t >
phys_addr_t, but you could try fixing those.

> I've tried the obvious experiment of just config'ing up a
> selection defining CONFIG_PHYS_ADDR_T_64BIT and seeing
> if things work.  They don't; details below.
> 
> So, a few questions...
> 
> Should using a 64-bit phys_addr_t on an A9 "just work"?  Or is
> this a new scenario never considered before?  Maybe a bug
> due to this odd configuration that might need to be fixed?
> Maybe some assumption of alignment, packed-ness
> or sizes in some structure relating to context switching?

It should mostly just work, since a lot of the same code gets
used by A7 and A15 when LPAE is enabled.

> I think the device-tree handling code is able to grok and process
> 64-bit addresses.  Does it make sense to allow the phys_addr_t and
> the resource_size_t to take on different sizes?  That is, to break the
> above definition of resource_size_t from types.h and allow its size
> to be determined independently of phys_addr_t?

resource_size_t and phys_addr_t should really be the same all the
time.

> Is there some easier (obvious even?) way to allow some DT devices to
> use 64-bit resources and I've just missed it?

You can try 64-bit dma_addr_t with 32-bit resource_size_t/phys_addr_t,
but it's more likely to work if you make them all 64-bit.

	Arnd

Some Large Address Space Ponders on A9

[Jon Loeliger]

On Tue, Jul 1, 2014 at 11:26 AM, Catalin Marinas
<catalin.marinas@arm.com> wrote:

>> Here's the problem.  Over in include/linux/types.h we find:
>>
>>     #ifdef CONFIG_PHYS_ADDR_T_64BIT
>>     typedef u64 phys_addr_t;
>>     #else
>>     typedef u32 phys_addr_t;
>>     #endif
>>
>>     typedef phys_addr_t resource_size_t;
>
> I won't comment on how you define this in DT but for the kernel things
> will likely break when you increase phys_addr_t to 64-bit on non-LPAE
> builds.
>
> First (only) question - how do you intend to map such physical address?
> The only way I'm aware of on A9 is using supersections and the ARM Linux
> port provides __arm_ioremap_pfn(). However, the supersections code is
> only enabled if !SMP.
>
> --
> Catalin

So, I don't really need to map in that as physical address space
that will be actually accessed.  I need to be able to parse the
large values out of the DT (2-cell values) into resource_size_t
variables and then manipulate the address space they represent.
Later, my IP block can handle the larger addresses, not as physical
memory, but as addresses.  I guess you could think of it as a
windowing or address translation mechanism that isn't coming
from the A9's 32-bit AXI bus.

Does that help explain the weird?  Or just confuse it? :-)

jdl

Some Large Address Space Ponders on A9

[Jon Loeliger]

Hi Arnd,

On Tue, Jul 1, 2014 at 11:43 AM, Arnd Bergmann <arnd@arndb.de> wrote:

> Hmm, we sometimes confuse phys_addr_t and dma_addr_t, because for
> all practical purposes they tend to be the same. However, I think
> what you need here is dma_addr_t, which is defined independently.
>
> You will probably run into bugs if you have dma_addr_t >
> phys_addr_t, but you could try fixing those.

Changing even dma_addr_t won't solve the issues that I see.
I need to be able to specify device tree devices that have 64-bit
resources.  All the subsequent math on those resources just falls
out and is written already in terms of resource_size_t data.

>> Should using a 64-bit phys_addr_t on an A9 "just work"?  Or is
>> this a new scenario never considered before?  Maybe a bug
>> due to this odd configuration that might need to be fixed?
>> Maybe some assumption of alignment, packed-ness
>> or sizes in some structure relating to context switching?
>
> It should mostly just work, since a lot of the same code gets
> used by A7 and A15 when LPAE is enabled.

Well, see, that's where I kind of thought we might be.  It seems
like the boot was working reasonably well.  But that maybe there
was some niggling "oh, duh!" bug that needed to be tracked down
in the context switch code or so...  And I just don't even begin to
understand enough of that black magic to start guessing yet. :-)

>> I think the device-tree handling code is able to grok and process
>> 64-bit addresses.  Does it make sense to allow the phys_addr_t and
>> the resource_size_t to take on different sizes?  That is, to break the
>> above definition of resource_size_t from types.h and allow its size
>> to be determined independently of phys_addr_t?
>
> resource_size_t and phys_addr_t should really be the same all the
> time.

Why?  What is the driving force or underlying assumption there?

>> Is there some easier (obvious even?) way to allow some DT devices to
>> use 64-bit resources and I've just missed it?
>
> You can try 64-bit dma_addr_t with 32-bit resource_size_t/phys_addr_t,
> but it's more likely to work if you make them all 64-bit.

Hmm.  So ponder enabling LPAE *except* for changes that it would
induce on page tables, MMU, addressing, illegal instructions, etc.
Just the address size changes, alignment, structure layout, etc.

World of hurt, right?

>         Arnd

Thanks,
jdl

Some Large Address Space Ponders on A9

[Jon Loeliger]

On Tue, Jul 1, 2014 at 11:43 AM, Arnd Bergmann <arnd@arndb.de> wrote:

>> I think the device-tree handling code is able to grok and process
>> 64-bit addresses.  Does it make sense to allow the phys_addr_t and
>> the resource_size_t to take on different sizes?  That is, to break the
>> above definition of resource_size_t from types.h and allow its size
>> to be determined independently of phys_addr_t?
>
> resource_size_t and phys_addr_t should really be the same all the
> time.
>
>         Arnd

So, as an experiment I did this:

diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index 1f8fed9..0dfb76f 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -631,6 +631,9 @@ config ARCH_PHYS_ADDR_T_64BIT
 config ARCH_DMA_ADDR_T_64BIT
        bool

+config RESOURCE_SIZE_T_64BIT
+       bool
+
 config ARM_THUMB
        bool "Support Thumb user binaries" if !CPU_THUMBONLY
        depends on CPU_ARM720T || CPU_ARM740T || CPU_ARM920T || CPU_ARM922T || \
diff --git a/include/linux/types.h b/include/linux/types.h
index 4d118ba..331d59a 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -164,7 +164,11 @@ typedef u64 phys_addr_t;
 typedef u32 phys_addr_t;
 #endif

+#ifdef CONFIG_RESOURCE_SIZE_T_64BIT
+typedef u64 resource_size_t;
+#else
 typedef phys_addr_t resource_size_t;
+#endif

And then this:

+       select RESOURCE_SIZE_T_64BIT

Verified a few things:

$ gdb -q ./vmlinux
Reading symbols from vmlinux...done.
(gdb) p sizeof(resource_size_t)
$1 = 8
(gdb) p sizeof(phys_addr_t)
$2 = 4
(gdb) p sizeof(dma_addr_t)
$3 = 4

And it all booted just fine.

jdl