arm64: iomem_resource doesn't contain all the region used

arm64: iomem_resource doesn't contain all the region used

[Julien Grall]

Hi all,

I've been working on trying to get balloon memory hotplug working for
ARM64 guest on Xen.

In order to find a suitable region to hotplug the fake memory, we are
trying to find a free region within iomem_resource.

But on ARM64, only an handful number of regions are listed in it. For
instance on X-Gene we have only:

42sh> cat /proc/iomem

10510000-105103ff : /soc/rtc at 10510000
1a400000-1a400fff : /soc/sata at 1a400000
1a800000-1a800fff : /soc/sata at 1a800000
1f220000-1f220fff : /soc/sata at 1a400000
1f227000-1f227fff : /soc/sata at 1a400000
1f22a000-1f22a0ff : /soc/phy at 1f22a000
1f22d000-1f22dfff : /soc/sata at 1a400000
1f22e000-1f22efff : /soc/sata at 1a400000
1f230000-1f230fff : /soc/sata at 1a800000
1f23a000-1f23a0ff : /soc/phy at 1f23a000
1f23d000-1f23dfff : /soc/sata at 1a800000
1f23e000-1f23efff : /soc/sata at 1a800000
1f2b0000-1f2bffff : csr
79000000-798fffff : /soc/msi at 79000000
4100000000-41ffffffff : System RAM
  4100080000-41008b58a3 : Kernel code
  410093c000-41009e9fff : Kernel data
e0d0000000-e0d003ffff : cfg

This is because most of the ARM drivers are using of_iomap which doesn't
register the region.

Looking to the code, I found a function of_io_request_and_map which
register the resource and does the mapping. I'm wondering if there is
any reason to have introduce a new function rather than doing the job in
of_iomap?

Although, that wouldn't fix all the drivers because some of them are
directly using ioremap. I've got in mind the initialization of GICv2 for
ACPI platform.

Is there any way we could register the IO region used on ARM without
having to enforce it in all the drivers?

Regards,

-- 
Julien Grall

arm64: iomem_resource doesn't contain all the region used

[Ian Campbell]

On Fri, 2015-10-23 at 15:58 +0100, Julien Grall wrote:
> Is there any way we could register the IO region used on ARM without
> having to enforce it in all the drivers?

This seems like an uphill battle to me.

Why not do as I suggested IRL yesterday and expose the map of "potential
RAM" addresses to the guest as well as the "actual RAM" addresses in the
regular memory properties.

i.e. explicitly expose the holes where RAM can be hotplugged later.

This is even analogous to a native memory hotplug case, which AIUI
similarly involves the provisioning of specific address space where RAM
might plausibly appear in the future (I don't think physical memory hotplug
involves searching for free PA space and hoping for the best, although
strange things have happened I guess).

With any luck you would be able to steal or define the bindings in terms of
the native hotplug case rather than inventing some Xen specific thing.

In terms of dom0 the "potential" RAM is the host's actual RAM banks.

In terms of domU the "potential" RAM is defined by the domain builder
layout (currently the two banks mentioned in Xen's arch-arm.h).

Ian.

arm64: iomem_resource doesn't contain all the region used

[Julien Grall]

(Adding David and Daniel)

On 23/10/15 16:45, Ian Campbell wrote:
> On Fri, 2015-10-23 at 15:58 +0100, Julien Grall wrote:
>> Is there any way we could register the IO region used on ARM without
>> having to enforce it in all the drivers?
> 
> This seems like an uphill battle to me.

I agree about it. However this is how x86 handle memory hotplug for xen
ballooning. I'm wondering how this is cannot an problem for x86?

Note that the problem is the same if a module is insert after hand.

> 
> Why not do as I suggested IRL yesterday and expose the map of "potential
> RAM" addresses to the guest as well as the "actual RAM" addresses in the
> regular memory properties.
> 
> i.e. explicitly expose the holes where RAM can be hotplugged later.

I was trying to find another solution because I find your suggestion
fragile.

Currently the device tree for a guest is set in stone after the creation
of the domain. I.e it's not possible to modify the device tree later
(I'm not speaking about hardcode value...).

This means that the region for "balloon hotplug" and "PCI hotplug" must
be static and can't overlapped. We may end up to run out of "PCI
hotplug" address space while there is plenty of free space in the
"balloon hotplug". However it's not possible to move from one to another.

How do you define the size of those regions? In one side, we can't
"hardcode" them because the user may not want to use either "balloon
hotplug" or "PCI hoplug". On another side, we could expose them to the
user but it's not nice.

> This is even analogous to a native memory hotplug case, which AIUI
> similarly involves the provisioning of specific address space where RAM
> might plausibly appear in the future (I don't think physical memory hotplug
> involves searching for free PA space and hoping for the best, although
> strange things have happened I guess).

I've looked at how Power PC handle native hotplug. From my
understanding, when a new memory bank is added, the device tree will be
updated by someone (firmware?) and an event will be sent to the Linux.

Linux will then read the new DT node (see
ibm,dynamic_reconfiguration-memory) and add the new memory region to Linux.

> 
> With any luck you would be able to steal or define the bindings in terms of
> the native hotplug case rather than inventing some Xen specific thing.

I wasn't able to find the binding for ibm,dynamic-reconfiguration-memory
in Linux.

> 
> In terms of dom0 the "potential" RAM is the host's actual RAM banks.

Your solution works for DOM0, but ...

> In terms of domU the "potential" RAM is defined by the domain builder
> layout (currently the two banks mentioned in Xen's arch-arm.h).

... the DOMU one is more complex (see above). Today the guest layout is
static, I wouldn't be surprised to see it becoming dynamic very soon (I
have in mind PCI hotplug) and therefore defining static hotplug region
would not possible.

Regards,

[1]
https://git.kernel.org/cgit/linux/kernel/git/xen/tip.git/commit/?h=for-linus-4.4&id=55b3da98a40dbb3776f7454daf0d95dde25c33d2


-- 
Julien Grall

arm64: iomem_resource doesn't contain all the region used

[Daniel Kiper]

On Wed, Oct 28, 2015 at 05:32:54PM +0000, Julien Grall wrote:
> (Adding David and Daniel)
>
> On 23/10/15 16:45, Ian Campbell wrote:
> > On Fri, 2015-10-23 at 15:58 +0100, Julien Grall wrote:
> >> Is there any way we could register the IO region used on ARM without
> >> having to enforce it in all the drivers?
> >
> > This seems like an uphill battle to me.
>
> I agree about it. However this is how x86 handle memory hotplug for xen
> ballooning. I'm wondering how this is cannot an problem for x86?
>
> Note that the problem is the same if a module is insert after hand.

Does ARM64 support memory hotplug on bare metal? If yes then check relevant
code and do what should be done as close as possible to bare metal case
on Xen guest.

> > Why not do as I suggested IRL yesterday and expose the map of "potential
> > RAM" addresses to the guest as well as the "actual RAM" addresses in the
> > regular memory properties.
> >
> > i.e. explicitly expose the holes where RAM can be hotplugged later.
>
> I was trying to find another solution because I find your suggestion
> fragile.
>
> Currently the device tree for a guest is set in stone after the creation
> of the domain. I.e it's not possible to modify the device tree later
> (I'm not speaking about hardcode value...).
>
> This means that the region for "balloon hotplug" and "PCI hotplug" must
> be static and can't overlapped. We may end up to run out of "PCI
> hotplug" address space while there is plenty of free space in the
> "balloon hotplug". However it's not possible to move from one to another.
>
> How do you define the size of those regions? In one side, we can't
> "hardcode" them because the user may not want to use either "balloon
> hotplug" or "PCI hoplug". On another side, we could expose them to the
> user but it's not nice.
>
> > This is even analogous to a native memory hotplug case, which AIUI
> > similarly involves the provisioning of specific address space where RAM
> > might plausibly appear in the future (I don't think physical memory hotplug
> > involves searching for free PA space and hoping for the best, although
> > strange things have happened I guess).
>
> I've looked at how Power PC handle native hotplug. From my
> understanding, when a new memory bank is added, the device tree will be
> updated by someone (firmware?) and an event will be sent to the Linux.
>
> Linux will then read the new DT node (see
> ibm,dynamic_reconfiguration-memory) and add the new memory region to Linux.

Make sense for me. It works more or less in the same way on bare metal x86.
Of course it uses ACPI instead of FDT.

> > With any luck you would be able to steal or define the bindings in terms of
> > the native hotplug case rather than inventing some Xen specific thing.
>
> I wasn't able to find the binding for ibm,dynamic-reconfiguration-memory
> in Linux.
>
> >
> > In terms of dom0 the "potential" RAM is the host's actual RAM banks.
>
> Your solution works for DOM0, but ...
>
> > In terms of domU the "potential" RAM is defined by the domain builder
> > layout (currently the two banks mentioned in Xen's arch-arm.h).
>
> ... the DOMU one is more complex (see above). Today the guest layout is
> static, I wouldn't be surprised to see it becoming dynamic very soon (I
> have in mind PCI hotplug) and therefore defining static hotplug region
> would not possible.

Please do not do that. I think that memory hotplug should not be limited
by anything but just a given platform limitations. By the way, could you
explain in details why linux/mm/memory_hotplug.c:register_memory_resource()
will not work on ARM64 guest?

Daniel

arm64: iomem_resource doesn't contain all the region used

[Julien Grall]

Hi Daniel,

On 29/10/15 16:36, Daniel Kiper wrote:
> On Wed, Oct 28, 2015 at 05:32:54PM +0000, Julien Grall wrote:
>> (Adding David and Daniel)
>>
>> On 23/10/15 16:45, Ian Campbell wrote:
>>> On Fri, 2015-10-23 at 15:58 +0100, Julien Grall wrote:
>>>> Is there any way we could register the IO region used on ARM without
>>>> having to enforce it in all the drivers?
>>>
>>> This seems like an uphill battle to me.
>>
>> I agree about it. However this is how x86 handle memory hotplug for xen
>> ballooning. I'm wondering how this is cannot an problem for x86?
>>
>> Note that the problem is the same if a module is insert after hand.
> 
> Does ARM64 support memory hotplug on bare metal? If yes then check relevant
> code and do what should be done as close as possible to bare metal case
> on Xen guest.

AFAICT, There is no support memory hotplug for ARM64 in Linux today.

[..]

>>> In terms of domU the "potential" RAM is defined by the domain builder
>>> layout (currently the two banks mentioned in Xen's arch-arm.h).
>>
>> ... the DOMU one is more complex (see above). Today the guest layout is
>> static, I wouldn't be surprised to see it becoming dynamic very soon (I
>> have in mind PCI hotplug) and therefore defining static hotplug region
>> would not possible.
> 
> Please do not do that. I think that memory hotplug should not be limited
> by anything but just a given platform limitations. By the way, could you
> explain in details why linux/mm/memory_hotplug.c:register_memory_resource()
> will not work on ARM64 guest?

Sorry I should have CCed you on the first mail where I explained the
problem.

The problem is not register_memory_resource but how the balloon code is
finding a free region in the address patch. With the patch [1] which
should land in Linux 4.4, the balloon code will look for a free region
within the iomem_resource. This means that we expect all the region used
(or will be used in the case the driver is loaded later) by a device are
registered.

However, on ARM64, only a handful of drivers are effectively registering
the I/O region.

Any drivers using directly ioremap* or of_iomap (the ioremap version
using the device tree node in parameter) won't register the I/O region used.

For instance on the board I'm using not even 10% of the I/O region are
registered:

42sh> cat /proc/iomem

10510000-105103ff : /soc/rtc at 10510000
1a400000-1a400fff : /soc/sata at 1a400000
1a800000-1a800fff : /soc/sata at 1a800000
1f220000-1f220fff : /soc/sata at 1a400000
1f227000-1f227fff : /soc/sata at 1a400000
1f22a000-1f22a0ff : /soc/phy at 1f22a000
1f22d000-1f22dfff : /soc/sata at 1a400000
1f22e000-1f22efff : /soc/sata at 1a400000
1f230000-1f230fff : /soc/sata at 1a800000
1f23a000-1f23a0ff : /soc/phy at 1f23a000
1f23d000-1f23dfff : /soc/sata at 1a800000
1f23e000-1f23efff : /soc/sata at 1a800000
1f2b0000-1f2bffff : csr
79000000-798fffff : /soc/msi at 79000000
4100000000-41ffffffff : System RAM
  4100080000-41008b58a3 : Kernel code
  410093c000-41009e9fff : Kernel data
e0d0000000-e0d003ffff : cfg

TBH I don't see why you don't hit this issue on x86. Overall some of the
drivers can be shared between the 2 architectures.

Regards,

[1]
https://git.kernel.org/cgit/linux/kernel/git/xen/tip.git/commit/?h=for-linus-4.4&id=55b3da98a40dbb3776f7454daf0d95dde25c33d2

-- 
Julien Grall

arm64: iomem_resource doesn't contain all the region used

[Daniel Kiper]

Hey Julien,

On Thu, Oct 29, 2015 at 05:24:42PM +0000, Julien Grall wrote:
> Hi Daniel,
>
> On 29/10/15 16:36, Daniel Kiper wrote:
> > On Wed, Oct 28, 2015 at 05:32:54PM +0000, Julien Grall wrote:
> >> (Adding David and Daniel)
> >>
> >> On 23/10/15 16:45, Ian Campbell wrote:
> >>> On Fri, 2015-10-23 at 15:58 +0100, Julien Grall wrote:
> >>>> Is there any way we could register the IO region used on ARM without
> >>>> having to enforce it in all the drivers?
> >>>
> >>> This seems like an uphill battle to me.
> >>
> >> I agree about it. However this is how x86 handle memory hotplug for xen
> >> ballooning. I'm wondering how this is cannot an problem for x86?
> >>
> >> Note that the problem is the same if a module is insert after hand.
> >
> > Does ARM64 support memory hotplug on bare metal? If yes then check relevant
> > code and do what should be done as close as possible to bare metal case
> > on Xen guest.
>
> AFAICT, There is no support memory hotplug for ARM64 in Linux today.

Are there any plans for it? Is anybody working on that stuff?

> >>> In terms of domU the "potential" RAM is defined by the domain builder
> >>> layout (currently the two banks mentioned in Xen's arch-arm.h).
> >>
> >> ... the DOMU one is more complex (see above). Today the guest layout is
> >> static, I wouldn't be surprised to see it becoming dynamic very soon (I
> >> have in mind PCI hotplug) and therefore defining static hotplug region
> >> would not possible.
> >
> > Please do not do that. I think that memory hotplug should not be limited
> > by anything but just a given platform limitations. By the way, could you
> > explain in details why linux/mm/memory_hotplug.c:register_memory_resource()
> > will not work on ARM64 guest?
>
> Sorry I should have CCed you on the first mail where I explained the
> problem.

No problem. Thanks for explanation.

> The problem is not register_memory_resource but how the balloon code is
> finding a free region in the address patch. With the patch [1] which
> should land in Linux 4.4, the balloon code will look for a free region
> within the iomem_resource. This means that we expect all the region used
> (or will be used in the case the driver is loaded later) by a device are
> registered.
>
> However, on ARM64, only a handful of drivers are effectively registering
> the I/O region.
>
> Any drivers using directly ioremap* or of_iomap (the ioremap version
> using the device tree node in parameter) won't register the I/O region used.
>
> For instance on the board I'm using not even 10% of the I/O region are
> registered:
>
> 42sh> cat /proc/iomem
>
> 10510000-105103ff : /soc/rtc at 10510000
> 1a400000-1a400fff : /soc/sata at 1a400000
> 1a800000-1a800fff : /soc/sata at 1a800000
> 1f220000-1f220fff : /soc/sata at 1a400000
> 1f227000-1f227fff : /soc/sata at 1a400000
> 1f22a000-1f22a0ff : /soc/phy at 1f22a000
> 1f22d000-1f22dfff : /soc/sata at 1a400000
> 1f22e000-1f22efff : /soc/sata at 1a400000
> 1f230000-1f230fff : /soc/sata at 1a800000
> 1f23a000-1f23a0ff : /soc/phy at 1f23a000
> 1f23d000-1f23dfff : /soc/sata at 1a800000
> 1f23e000-1f23efff : /soc/sata at 1a800000
> 1f2b0000-1f2bffff : csr
> 79000000-798fffff : /soc/msi at 79000000
> 4100000000-41ffffffff : System RAM
>   4100080000-41008b58a3 : Kernel code
>   410093c000-41009e9fff : Kernel data
> e0d0000000-e0d003ffff : cfg

Ugh! I though that it is a requirement that every memory/io region user
must register it using relevant function. It looks that it is not true.
So, there is only one reliable way to get info about used io/memory regions.
You must look at DT. However, if driver may agree with a device other
config and move used io/memory regions to different place without updating
DT then we are lost.

> TBH I don't see why you don't hit this issue on x86. Overall some of the
> drivers can be shared between the 2 architectures.

Are you able to point out any (x86) driver which does not behave as it should?

Daniel

arm64: iomem_resource doesn't contain all the region used

[Julien Grall]

On 30/10/15 17:53, Daniel Kiper wrote:
> Hey Julien,

Hi,

> On Thu, Oct 29, 2015 at 05:24:42PM +0000, Julien Grall wrote:
>> Hi Daniel,
>>
>> On 29/10/15 16:36, Daniel Kiper wrote:
>>> On Wed, Oct 28, 2015 at 05:32:54PM +0000, Julien Grall wrote:
>>>> (Adding David and Daniel)
>>>>
>>>> On 23/10/15 16:45, Ian Campbell wrote:
>>>>> On Fri, 2015-10-23 at 15:58 +0100, Julien Grall wrote:
>>>>>> Is there any way we could register the IO region used on ARM without
>>>>>> having to enforce it in all the drivers?
>>>>>
>>>>> This seems like an uphill battle to me.
>>>>
>>>> I agree about it. However this is how x86 handle memory hotplug for xen
>>>> ballooning. I'm wondering how this is cannot an problem for x86?
>>>>
>>>> Note that the problem is the same if a module is insert after hand.
>>>
>>> Does ARM64 support memory hotplug on bare metal? If yes then check relevant
>>> code and do what should be done as close as possible to bare metal case
>>> on Xen guest.
>>
>> AFAICT, There is no support memory hotplug for ARM64 in Linux today.
> 
> Are there any plans for it? Is anybody working on that stuff?

I'm not aware of any plan. But I started to look at it and adding
arch_add_memory (the arch-specific function required to support memory
hotplug) should be pretty easy. It's a matter of few lines of code.

> 
>>>>> In terms of domU the "potential" RAM is defined by the domain builder
>>>>> layout (currently the two banks mentioned in Xen's arch-arm.h).
>>>>
>>>> ... the DOMU one is more complex (see above). Today the guest layout is
>>>> static, I wouldn't be surprised to see it becoming dynamic very soon (I
>>>> have in mind PCI hotplug) and therefore defining static hotplug region
>>>> would not possible.
>>>
>>> Please do not do that. I think that memory hotplug should not be limited
>>> by anything but just a given platform limitations. By the way, could you
>>> explain in details why linux/mm/memory_hotplug.c:register_memory_resource()
>>> will not work on ARM64 guest?
>>
>> Sorry I should have CCed you on the first mail where I explained the
>> problem.
> 
> No problem. Thanks for explanation.
> 
>> The problem is not register_memory_resource but how the balloon code is
>> finding a free region in the address patch. With the patch [1] which
>> should land in Linux 4.4, the balloon code will look for a free region
>> within the iomem_resource. This means that we expect all the region used
>> (or will be used in the case the driver is loaded later) by a device are
>> registered.
>>
>> However, on ARM64, only a handful of drivers are effectively registering
>> the I/O region.
>>
>> Any drivers using directly ioremap* or of_iomap (the ioremap version
>> using the device tree node in parameter) won't register the I/O region used.
>>
>> For instance on the board I'm using not even 10% of the I/O region are
>> registered:
>>
>> 42sh> cat /proc/iomem
>>
>> 10510000-105103ff : /soc/rtc at 10510000
>> 1a400000-1a400fff : /soc/sata at 1a400000
>> 1a800000-1a800fff : /soc/sata at 1a800000
>> 1f220000-1f220fff : /soc/sata at 1a400000
>> 1f227000-1f227fff : /soc/sata at 1a400000
>> 1f22a000-1f22a0ff : /soc/phy at 1f22a000
>> 1f22d000-1f22dfff : /soc/sata at 1a400000
>> 1f22e000-1f22efff : /soc/sata at 1a400000
>> 1f230000-1f230fff : /soc/sata at 1a800000
>> 1f23a000-1f23a0ff : /soc/phy at 1f23a000
>> 1f23d000-1f23dfff : /soc/sata at 1a800000
>> 1f23e000-1f23efff : /soc/sata at 1a800000
>> 1f2b0000-1f2bffff : csr
>> 79000000-798fffff : /soc/msi at 79000000
>> 4100000000-41ffffffff : System RAM
>>   4100080000-41008b58a3 : Kernel code
>>   410093c000-41009e9fff : Kernel data
>> e0d0000000-e0d003ffff : cfg
> 
> Ugh! I though that it is a requirement that every memory/io region user
> must register it using relevant function. It looks that it is not true.
> So, there is only one reliable way to get info about used io/memory regions.
> You must look at DT. However, if driver may agree with a device other
> config and move used io/memory regions to different place without updating
> DT then we are lost.

While the Linux folks are trying to describe all the device in the
Device Tree, it not always the case.

Also, browsing the device tree to find memory range is a pain and quite
fragile. For instance we already do that in the hypervisor to map all
the device to DOM0 (see arch/arm/domain_build.c) but we still do have
bug report of platform not working with this solution.

> 
>> TBH I don't see why you don't hit this issue on x86. Overall some of the
>> drivers can be shared between the 2 architectures.
> 
> Are you able to point out any (x86) driver which does not behave as it should?

Just thinking that on x86 you have the e820 which describe the memory
layout of the platform.

Am I correct to say that every I/O regions are described in the e820 and
therefore registered when Linux is booting?

Regards,

-- 
Julien Grall

arm64: iomem_resource doesn't contain all the region used

[Daniel Kiper]

On Fri, Oct 30, 2015 at 06:32:54PM +0000, Julien Grall wrote:

[...]

> > Ugh! I though that it is a requirement that every memory/io region user
> > must register it using relevant function. It looks that it is not true.
> > So, there is only one reliable way to get info about used io/memory regions.
> > You must look at DT. However, if driver may agree with a device other
> > config and move used io/memory regions to different place without updating
> > DT then we are lost.
>
> While the Linux folks are trying to describe all the device in the
> Device Tree, it not always the case.
>
> Also, browsing the device tree to find memory range is a pain and quite
> fragile. For instance we already do that in the hypervisor to map all
> the device to DOM0 (see arch/arm/domain_build.c) but we still do have
> bug report of platform not working with this solution.

I expected this. Do we have any other choice? Maybe we should discuss
this on wider forum somewhere and encourage devs to use relevant resource
management functions in drivers. Well, I saw somewhere that IanC doubts
it is feasible but I do not see better solution for this right now.

> >> TBH I don't see why you don't hit this issue on x86. Overall some of the
> >> drivers can be shared between the 2 architectures.
> >
> > Are you able to point out any (x86) driver which does not behave as it should?
>
> Just thinking that on x86 you have the e820 which describe the memory
> layout of the platform.
>
> Am I correct to say that every I/O regions are described in the e820 and
> therefore registered when Linux is booting?

No, as I saw regions used by devices are not listed on some (all?) machines.
I have a feeling that driver negotiates with device where a given I/O
region should be placed. So, this region is not known by BIOS or even UEFI
in advance. Sadly, I do not know in detail how this process works.

Daniel