Fail to start Dom0 in ARM highmem.

Fail to start Dom0 in ARM highmem.

[Iurii Konovalenko]

Hello, all!

I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
Xen revision is 4.4.
I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
This SoC support ARM LPAE. Physical address is 40-bit long.
Board has 4GB of RAM. Adress Map is following:
0x00_4000_0000 - 0x00_7FFF_FFFF 1 GBytes of RAM
0x01_4000_0000 - 0x01_FFFF_FFFF 3 GBytes of RAM
I want to provide 1 GB of memory for Dom0.
As there is not enough memory for Dom0 in first bank of memory, it is
allocated by Xen from second bank on pa 0x01_4000_0000 -
0x01_7FFF_FFFF.
And in context of this task I've faced with problem. Here is log print:
>(XEN) Populate P2M 0x140000000->0x180000000 (1:1 mapping for dom0)
>(XEN) Loading kernel from boot module 2
>(XEN) Loading zImage from 0000000072000040 to 0000000140008000-0000000140327aa8
>(XEN)
>(XEN) ****************************************
>(XEN) Panic on CPU 0:
>(XEN) Unable to map translate guest address
>(XEN) ****************************************

My investigations show, that it is because of type cast truncating.
In file xen/arch/arm/kernel.c in line 152 we call
    rc = gvirt_to_maddr(load_addr + offs, &ma);
where load_addr has type paddr_t (64-bit), in my case
0x0000000140008000, but function deceleration is
    static inline int gvirt_to_maddr(vaddr_t va, paddr_t *pa)
where va has type vaddr_t (32-bit), in my case is truncated to 0x40008000
That's why bits 33-40 are lost and function fails.

Could you please provide any suggestions, is it possible to start Dom0
with 1:1 mapping and memory in second bank (over32-bit high memory)?

Best regards.

Iurii Konovalenko | Senior Software Engineer
GlobalLogic

Re: Fail to start Dom0 in ARM highmem.

[Julien Grall]

On 10/28/2014 03:47 PM, Iurii Konovalenko wrote:
> Hello, all!

Hello Iurii,

> I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
> Xen revision is 4.4.
> I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
> This SoC support ARM LPAE. Physical address is 40-bit long.
> Board has 4GB of RAM. Adress Map is following:
> 0x00_4000_0000 - 0x00_7FFF_FFFF 1 GBytes of RAM
> 0x01_4000_0000 - 0x01_FFFF_FFFF 3 GBytes of RAM
> I want to provide 1 GB of memory for Dom0.
> As there is not enough memory for Dom0 in first bank of memory, it is
> allocated by Xen from second bank on pa 0x01_4000_0000 -
> 0x01_7FFF_FFFF.
> And in context of this task I've faced with problem. Here is log print:
>> (XEN) Populate P2M 0x140000000->0x180000000 (1:1 mapping for dom0)
>> (XEN) Loading kernel from boot module 2
>> (XEN) Loading zImage from 0000000072000040 to 0000000140008000-0000000140327aa8
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Unable to map translate guest address
>> (XEN) ****************************************
> 
> My investigations show, that it is because of type cast truncating.
> In file xen/arch/arm/kernel.c in line 152 we call
>     rc = gvirt_to_maddr(load_addr + offs, &ma);
> where load_addr has type paddr_t (64-bit), in my case
> 0x0000000140008000, but function deceleration is
>     static inline int gvirt_to_maddr(vaddr_t va, paddr_t *pa)
> where va has type vaddr_t (32-bit), in my case is truncated to 0x40008000
> That's why bits 33-40 are lost and function fails.
> 
> Could you please provide any suggestions, is it possible to start Dom0
> with 1:1 mapping and memory in second bank (over32-bit high memory)?

32-bit processor is only able to address 32 bits virtual address. When
pagination is disabled, only 32 bits physical address will work.

Therefore DOM0 should have at least few MB of RAM below 4GB, where the
kernel should be loaded.

I suspect Xen is not able to find free space below 4G, because your
bootloader is loading the kernel, initramfs... in this first bank. Can
you teach your bootloader to load them above 4G to free the space?

Although, Xen should throw a warning on 32bit Xen is we are unable to
find space below 4G. I will look to send a patch for this purpose.

Regards,

-- 
Julien Grall

Re: Fail to start Dom0 in ARM highmem.

[Ian Campbell]

On Tue, 2014-10-28 at 17:13 +0000, Julien Grall wrote:
> On 10/28/2014 03:47 PM, Iurii Konovalenko wrote:
> > Hello, all!
> 
> Hello Iurii,
> 
> > I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
> > Xen revision is 4.4.
> > I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
> > This SoC support ARM LPAE. Physical address is 40-bit long.
> > Board has 4GB of RAM. Adress Map is following:
> > 0x00_4000_0000 - 0x00_7FFF_FFFF 1 GBytes of RAM
> > 0x01_4000_0000 - 0x01_FFFF_FFFF 3 GBytes of RAM
> > I want to provide 1 GB of memory for Dom0.
> > As there is not enough memory for Dom0 in first bank of memory, it is
> > allocated by Xen from second bank on pa 0x01_4000_0000 -
> > 0x01_7FFF_FFFF.
> > And in context of this task I've faced with problem. Here is log print:
> >> (XEN) Populate P2M 0x140000000->0x180000000 (1:1 mapping for dom0)
> >> (XEN) Loading kernel from boot module 2
> >> (XEN) Loading zImage from 0000000072000040 to 0000000140008000-0000000140327aa8
> >> (XEN)
> >> (XEN) ****************************************
> >> (XEN) Panic on CPU 0:
> >> (XEN) Unable to map translate guest address
> >> (XEN) ****************************************
> > 
> > My investigations show, that it is because of type cast truncating.
> > In file xen/arch/arm/kernel.c in line 152 we call
> >     rc = gvirt_to_maddr(load_addr + offs, &ma);
> > where load_addr has type paddr_t (64-bit), in my case
> > 0x0000000140008000, but function deceleration is
> >     static inline int gvirt_to_maddr(vaddr_t va, paddr_t *pa)
> > where va has type vaddr_t (32-bit), in my case is truncated to 0x40008000
> > That's why bits 33-40 are lost and function fails.
> > 
> > Could you please provide any suggestions, is it possible to start Dom0
> > with 1:1 mapping and memory in second bank (over32-bit high memory)?
> 
> 32-bit processor is only able to address 32 bits virtual address. When
> pagination is disabled, only 32 bits physical address will work.
> 
> Therefore DOM0 should have at least few MB of RAM below 4GB, where the
> kernel should be loaded.
> 
> I suspect Xen is not able to find free space below 4G, because your
> bootloader is loading the kernel, initramfs... in this first bank. Can
> you teach your bootloader to load them above 4G to free the space?
> 
> Although, Xen should throw a warning on 32bit Xen is we are unable to
> find space below 4G. I will look to send a patch for this purpose.

Xen 4.5 ought to use multiple banks to give as much RAM below 4G as it
can and then satisfy the rest from highmem. It would be interesting to
see how it behaves in this setup.

Ian.

Re: Fail to start Dom0 in ARM highmem.

[Andrii Anisov]

> Xen 4.5 ought to use multiple banks to give as much RAM below 4G as it
> can and then satisfy the rest from highmem. It would be interesting to
> see how it behaves in this setup.

We did just the first attempt with our 4.4 as well know stuff.
We will go with 4.5 further.

Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664  M +380505738852  S andriyanisov
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: Fail to start Dom0 in ARM highmem.

[Ian Campbell]

On Wed, 2014-10-29 at 19:35 +0200, Andrii Anisov wrote:
> > Xen 4.5 ought to use multiple banks to give as much RAM below 4G as it
> > can and then satisfy the rest from highmem. It would be interesting to
> > see how it behaves in this setup.
> 
> We did just the first attempt with our 4.4 as well know stuff.
> We will go with 4.5 further.

Good plan.

If you wanted to stick with 4.4 my recommendation would be to either use
less RAM for dom0 or to add some sort of ugly hack to split the RAM into
two banks (e.g. 0.5G in each, one low one high).

Ian.

Re: Fail to start Dom0 in ARM highmem.

[Andrii Anisov]

> If you wanted to stick with 4.4

No we definitely will switch to 4.5.
We plan to bringup driver domain on that setup, so will need stuff
from 4.5. Like iomem/interrupt assignment for domU, we will bringup
their IPMMU (close to ARM SMMU but different), etc.

> either use less RAM for dom0

Yep, that is what we actually will do anyway. We estimate dom0 as a
"thin dom0". I assume with block device driver only, so it will not
need lot of RAM.

Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664  M +380505738852  S andriyanisov
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: Fail to start Dom0 in ARM highmem.

[Iurii Konovalenko]

Hi, all!

At last, I started my setup on Xen 4.5.
Xen has successfully allocated 1GB for Dom0, splitting it in two banks:

(XEN) Allocating 1:1 mappings totalling 1024MB for dom0:
(XEN) BANK[0] 0x00000040000000-0x00000070000000 (768MB)
(XEN) BANK[1] 0x000001c0000000-0x000001d0000000 (256MB)

Thanks a lot guys for your support!
Best regards.

Iurii Konovalenko | Senior Software Engineer
GlobalLogic
P +3.8044.492.9695 M +38.099.932.2909
S yufuntik
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt


On Thu, Oct 30, 2014 at 1:19 PM, Andrii Anisov
<andrii.anisov@globallogic.com> wrote:
>> If you wanted to stick with 4.4
>
> No we definitely will switch to 4.5.
> We plan to bringup driver domain on that setup, so will need stuff
> from 4.5. Like iomem/interrupt assignment for domU, we will bringup
> their IPMMU (close to ARM SMMU but different), etc.
>
>> either use less RAM for dom0
>
> Yep, that is what we actually will do anyway. We estimate dom0 as a
> "thin dom0". I assume with block device driver only, so it will not
> need lot of RAM.
>
> Andrii Anisov | Team Lead
> GlobalLogic
> Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
> P +38.044.492.9695x3664  M +380505738852  S andriyanisov
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt