[PATCH 0/5] aarch64, common: improve clang and llvm support

[PATCH 0/5] aarch64, common: improve clang and llvm support

[Volodymyr Babchuk]

This patch series addresses two main issues:

1. Some errors when using clang for aarch64 build

2. Unability to use clang for cross-compilation

While first problem has straightforward fix: add -march argument and
allow to use FP registers for vfg.c, second one is much trickier. I am
aware of the earlier attempts (like [1]), so I decided to use
different approach. Bscailly, I chose to use LLVM tooling as an
option. As LLVM tooling supports cross-compilation inherently, we
don't need to use CROSS_COMPILE varialbe at all.

  make XEN_TARGET_ARCH=arm64 clang=y llvm=y

Shoud work on all host platforms, although I tested this only on x86.

[1] https://patchwork.kernel.org/project/xen-devel/cover/20190327184531.30986-1-julien.grall@arm.com/

Volodymyr Babchuk (5):
  arm: bugframe: emit msg offset only if msg is present
  build: add possibility to use LLVM tools
  build: arm64: provide -target and -march if using clang
  xen: build: add support for CFLAGS_REMOVE variable
  xen: arm64: remove -mgeneral-regs-only for vfp.c

 config/GNUCommon.mk            | 16 ++++++++++++++++
 config/Linux.mk                |  4 ++++
 config/StdGNU.mk               | 18 +-----------------
 config/arm64.mk                |  4 ++++
 config/llvm.mk                 | 17 +++++++++++++++++
 xen/Rules.mk                   |  2 +-
 xen/arch/arm/arm64/Makefile    |  2 ++
 xen/arch/arm/include/asm/bug.h |  4 ++++
 8 files changed, 49 insertions(+), 18 deletions(-)
 create mode 100644 config/GNUCommon.mk
 create mode 100644 config/llvm.mk

-- 
2.47.0

[PATCH 1/5] arm: bugframe: emit msg offset only if msg is present

[Volodymyr Babchuk]

Before this change, compiler would generate symbol that points right
past .rodata.str section. While GNU ld handles this just fine, LLVM ld
will complain:

ld.lld: error: common/device-tree/device-tree.o:(.rodata.str): offset is outside the section

Fix this issue by providing literal zero instead of calculated zero if
there is no error message in a bugframe.

Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
 xen/arch/arm/include/asm/bug.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/xen/arch/arm/include/asm/bug.h b/xen/arch/arm/include/asm/bug.h
index cacaf014ab..9339e3bbf6 100644
--- a/xen/arch/arm/include/asm/bug.h
+++ b/xen/arch/arm/include/asm/bug.h
@@ -47,7 +47,11 @@ struct bug_frame {
          ".p2align 2\n"                                                     \
          ".long (1b - 4b)\n"                                                \
          ".long (2b - 4b)\n"                                                \
+         ".if " #has_msg "\n"                                               \
          ".long (3b - 4b)\n"                                                \
+         ".else\n"                                                          \
+         ".long 0\n"                                                        \
+         ".endif\n"                                                         \
          ".hword " __stringify(line) ", 0\n"                                \
          ".popsection");                                                    \
 } while (0)
-- 
2.47.0

Re: [PATCH 1/5] arm: bugframe: emit msg offset only if msg is present

[Jan Beulich]

On 29.11.2024 02:49, Volodymyr Babchuk wrote:
> Before this change, compiler would generate symbol that points right
> past .rodata.str section. While GNU ld handles this just fine, LLVM ld
> will complain:
> 
> ld.lld: error: common/device-tree/device-tree.o:(.rodata.str): offset is outside the section

As this is pretty clearly a bug in the linker, has that been reported
there? A reference to such a bug report would then imo want ...

> --- a/xen/arch/arm/include/asm/bug.h
> +++ b/xen/arch/arm/include/asm/bug.h
> @@ -47,7 +47,11 @@ struct bug_frame {
>           ".p2align 2\n"                                                     \
>           ".long (1b - 4b)\n"                                                \
>           ".long (2b - 4b)\n"                                                \
> +         ".if " #has_msg "\n"                                               \
>           ".long (3b - 4b)\n"                                                \
> +         ".else\n"                                                          \
> +         ".long 0\n"                                                        \
> +         ".endif\n"                                                         \
>           ".hword " __stringify(line) ", 0\n"                                \
>           ".popsection");                                                    \
>  } while (0)

... attaching as a comment here, to make clear why the extra complexity is
needed.

Jan

[PATCH 2/5] build: add possibility to use LLVM tools

[Volodymyr Babchuk]

Currently, even if we are using clang as a C compiler, we still use
GNU binutils. This patch adds new option "llvm" that allows to use
linker, objcopy and all other tools from LLVM project. As LLVM tools
use different approach for cross-compilation, we don't need
CROSS_COMPILE prefix in this case.

Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
 config/GNUCommon.mk | 16 ++++++++++++++++
 config/Linux.mk     |  4 ++++
 config/StdGNU.mk    | 18 +-----------------
 config/llvm.mk      | 17 +++++++++++++++++
 4 files changed, 38 insertions(+), 17 deletions(-)
 create mode 100644 config/GNUCommon.mk
 create mode 100644 config/llvm.mk

diff --git a/config/GNUCommon.mk b/config/GNUCommon.mk
new file mode 100644
index 0000000000..71c2303166
--- /dev/null
+++ b/config/GNUCommon.mk
@@ -0,0 +1,16 @@
+# Allow git to be wrappered in the environment
+GIT        ?= git
+
+INSTALL      = install
+INSTALL_DIR  = $(INSTALL) -d -m0755 -p
+INSTALL_DATA = $(INSTALL) -m0644 -p
+INSTALL_PROG = $(INSTALL) -m0755 -p
+
+BOOT_DIR ?= /boot
+DEBUG_DIR ?= /usr/lib/debug
+
+SOCKET_LIBS =
+UTIL_LIBS = -lutil
+
+SONAME_LDFLAG = -soname
+SHLIB_LDFLAGS = -shared
diff --git a/config/Linux.mk b/config/Linux.mk
index 2a84b6b0f3..6f4dc865a2 100644
--- a/config/Linux.mk
+++ b/config/Linux.mk
@@ -1,3 +1,7 @@
+ifeq ($(llvm),y)
+include $(XEN_ROOT)/config/llvm.mk
+else
 include $(XEN_ROOT)/config/StdGNU.mk
+endif
 
 SYSCONFIG_DIR = $(CONFIG_DIR)/$(CONFIG_LEAF_DIR)
diff --git a/config/StdGNU.mk b/config/StdGNU.mk
index aaa0d007f7..d723bc274e 100644
--- a/config/StdGNU.mk
+++ b/config/StdGNU.mk
@@ -19,20 +19,4 @@ OBJCOPY    = $(CROSS_COMPILE)objcopy
 OBJDUMP    = $(CROSS_COMPILE)objdump
 SIZEUTIL   = $(CROSS_COMPILE)size
 
-# Allow git to be wrappered in the environment
-GIT        ?= git
-
-INSTALL      = install
-INSTALL_DIR  = $(INSTALL) -d -m0755 -p
-INSTALL_DATA = $(INSTALL) -m0644 -p
-INSTALL_PROG = $(INSTALL) -m0755 -p
-
-BOOT_DIR ?= /boot
-DEBUG_DIR ?= /usr/lib/debug
-
-SOCKET_LIBS =
-UTIL_LIBS = -lutil
-
-SONAME_LDFLAG = -soname
-SHLIB_LDFLAGS = -shared
-
+include $(XEN_ROOT)/config/GNUCommon.mk
diff --git a/config/llvm.mk b/config/llvm.mk
new file mode 100644
index 0000000000..e474428286
--- /dev/null
+++ b/config/llvm.mk
@@ -0,0 +1,17 @@
+AS         = llvm-as
+LD         = ld.lld
+CC         = clang
+CXX        = clang++
+LD_LTO     = llvm-lto
+CPP        = $(CC) -E
+ADDR2LINE  = llvm-addr2line
+AR         = llvm-ar
+RANLIB     = llvm-ranlib
+NM         = llvm-nm
+STRIP      = llvm-strip
+OBJCOPY    = llvm-objcopy
+OBJDUMP    = llvm-objdump
+SIZEUTIL   = llvm-size
+
+include $(XEN_ROOT)/config/GNUCommon.mk
+
-- 
2.47.0

Re: [PATCH 2/5] build: add possibility to use LLVM tools

[Jan Beulich]

On 29.11.2024 02:49, Volodymyr Babchuk wrote:
> Currently, even if we are using clang as a C compiler, we still use
> GNU binutils. This patch adds new option "llvm" that allows to use
> linker, objcopy and all other tools from LLVM project. As LLVM tools
> use different approach for cross-compilation, we don't need
> CROSS_COMPILE prefix in this case.

This new option is meant to control both toolstack and hypervisor builds?
As to the latter, I assume you're aware we're trying to move away from
this kind of command line control of the build.

How is this intended to interact with the existing $(clang) that we have?
In the cover letter you appear to only care about the clang=y llvm=y case,
while ...

> --- a/config/StdGNU.mk
> +++ b/config/StdGNU.mk
> @@ -19,20 +19,4 @@ OBJCOPY    = $(CROSS_COMPILE)objcopy
>  OBJDUMP    = $(CROSS_COMPILE)objdump
>  SIZEUTIL   = $(CROSS_COMPILE)size

... even up from here there are uses of $(clang).

Also please Cc Anthony on build system changes.

Jan

Re: [PATCH 2/5] build: add possibility to use LLVM tools

[Anthony PERARD]

On Fri, Nov 29, 2024 at 08:57:47AM +0100, Jan Beulich wrote:
> On 29.11.2024 02:49, Volodymyr Babchuk wrote:
> > Currently, even if we are using clang as a C compiler, we still use
> > GNU binutils. This patch adds new option "llvm" that allows to use
> > linker, objcopy and all other tools from LLVM project. As LLVM tools
> > use different approach for cross-compilation, we don't need
> > CROSS_COMPILE prefix in this case.

It might be worth explaining what this different approach is ;-). My
guess is that you ask llvm/clang to build for a specific arch, via
XEN_TARGET_ARCH=riscv64, and llvm doesn't need different binaries will
just do what is needed, right? (with -march I guess).

> This new option is meant to control both toolstack and hypervisor builds?
> As to the latter, I assume you're aware we're trying to move away from
> this kind of command line control of the build.

Having "clang=y" is a bit useless when one can do "CC=clang" and let
the build system figure out what CC is. That's at least true for the
build system under "xen/".

But if one want to use the whole LLVM toolchain, it as to write a lot
more. To build Xen (just the hypervisor) with it, one would need to run:

make CC=clang CXX=clang++ AS=llvm-as LD=ld.lld LD_LTO=llvm-lto \
    ADDR2LINE=llvm-addr2line AR=llvm-ar RANLIB=llvm-ranlib NM=llvm-nm \
    STRIP=llvm-strip OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump \
    SIZEUTIL=llvm-size

So while it's possible, maybe introducing "llvm" or "LLVM" to switch
toolchain might be ok, to help. (Just because Linux did just that
recently:
    https://www.kernel.org/doc/html/latest/kbuild/llvm.html#the-llvm-argument
)

Beyond switching toolchain, I don't think $(llvm) or $(clang) should be
used in the build system, and instead rely on autodetection for
arguments. (There's already $(CONFIG_CC_IS_CLANG) and
$(CONFIG_LD_IS_LLVM) in the hypervisor's build system, via Kconfig)

At least for the hypervisor. For the toolstack, we should probably deal
with toolchain in ./configure.

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[PATCH 3/5] build: arm64: provide -target and -march if using clang

[Volodymyr Babchuk]

Provide -target and -march explicitly when building with clang. This
makes cross-compilation much easier, because clang accept this
parameters regardless of host platform. Basically,

  make XEN_TARGET_ARCH=arm64 clang=y llvm=y

will behave in the same way if building Xen on x86, or on arm64 or on
any other platform.

-march is required because with default value, clang will not
recognize EL2 registers.

Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
 config/arm64.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/config/arm64.mk b/config/arm64.mk
index c4662f67d0..97eb9a82e7 100644
--- a/config/arm64.mk
+++ b/config/arm64.mk
@@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
 
 CFLAGS += #-marm -march= -mcpu= etc
 
+ifeq ($(clang),y)
+CFLAGS += -target aarch64 -march=armv8-a
+endif
+
 # Use only if calling $(LD) directly.
 LDFLAGS_DIRECT += -EL
 
-- 
2.47.0

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Jan Beulich]

On 29.11.2024 02:49, Volodymyr Babchuk wrote:
> Provide -target and -march explicitly when building with clang. This
> makes cross-compilation much easier, because clang accept this
> parameters regardless of host platform. Basically,
> 
>   make XEN_TARGET_ARCH=arm64 clang=y llvm=y
> 
> will behave in the same way if building Xen on x86, or on arm64 or on
> any other platform.
> 
> -march is required because with default value, clang will not
> recognize EL2 registers.
> 
> Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
> ---
>  config/arm64.mk | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/config/arm64.mk b/config/arm64.mk
> index c4662f67d0..97eb9a82e7 100644
> --- a/config/arm64.mk
> +++ b/config/arm64.mk
> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>  
>  CFLAGS += #-marm -march= -mcpu= etc
>  
> +ifeq ($(clang),y)
> +CFLAGS += -target aarch64 -march=armv8-a
> +endif

Why is this dependent on (just?) $(clang), not (also?) $(llvm)? Also
this affects both toolstack builds and hypervisor. Is applying -march
like this actually appropriate for the toolstack?

Jan

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Volodymyr Babchuk]

Hi Jan,

Jan Beulich <jbeulich@suse.com> writes:

> On 29.11.2024 02:49, Volodymyr Babchuk wrote:
>> Provide -target and -march explicitly when building with clang. This
>> makes cross-compilation much easier, because clang accept this
>> parameters regardless of host platform. Basically,
>> 
>>   make XEN_TARGET_ARCH=arm64 clang=y llvm=y
>> 
>> will behave in the same way if building Xen on x86, or on arm64 or on
>> any other platform.
>> 
>> -march is required because with default value, clang will not
>> recognize EL2 registers.
>> 
>> Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>> ---
>>  config/arm64.mk | 4 ++++
>>  1 file changed, 4 insertions(+)
>> 
>> diff --git a/config/arm64.mk b/config/arm64.mk
>> index c4662f67d0..97eb9a82e7 100644
>> --- a/config/arm64.mk
>> +++ b/config/arm64.mk
>> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>>  
>>  CFLAGS += #-marm -march= -mcpu= etc
>>  
>> +ifeq ($(clang),y)
>> +CFLAGS += -target aarch64 -march=armv8-a
>> +endif
>
> Why is this dependent on (just?) $(clang), not (also?) $(llvm)?

Because this parameter is handled by clang only. There is no harm in
providing it explicitly. When building on arm64, value of this parameter
will match the default value for the platform. When building on x86, we
need to tell clang that it should generate arm64 code anyways. There is
no reason in trying to make ARM build with x86 instruction set.

> Also
> this affects both toolstack builds and hypervisor. Is applying -march
> like this actually appropriate for the toolstack?

This is a good question. I can't see why this can't be appropriate for
toolstack. I.e. what bad can happen when building the toolstack.

-- 
WBR, Volodymyr

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Julien Grall]

Hi,

On 29/11/2024 22:12, Volodymyr Babchuk wrote:
> 
> Hi Jan,
> 
> Jan Beulich <jbeulich@suse.com> writes:
> 
>> On 29.11.2024 02:49, Volodymyr Babchuk wrote:
>>> Provide -target and -march explicitly when building with clang. This
>>> makes cross-compilation much easier, because clang accept this
>>> parameters regardless of host platform. Basically,
>>>
>>>    make XEN_TARGET_ARCH=arm64 clang=y llvm=y
>>>
>>> will behave in the same way if building Xen on x86, or on arm64 or on
>>> any other platform.
>>>
>>> -march is required because with default value, clang will not
>>> recognize EL2 registers.
>>>
>>> Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>>> ---
>>>   config/arm64.mk | 4 ++++
>>>   1 file changed, 4 insertions(+)
>>>
>>> diff --git a/config/arm64.mk b/config/arm64.mk
>>> index c4662f67d0..97eb9a82e7 100644
>>> --- a/config/arm64.mk
>>> +++ b/config/arm64.mk
>>> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>>>   
>>>   CFLAGS += #-marm -march= -mcpu= etc
>>>   
>>> +ifeq ($(clang),y)
>>> +CFLAGS += -target aarch64 -march=armv8-a
>>> +endif
>>
>> Why is this dependent on (just?) $(clang), not (also?) $(llvm)?
> 
> Because this parameter is handled by clang only. There is no harm in
> providing it explicitly. When building on arm64, value of this parameter
> will match the default value for the platform. When building on x86, we
> need to tell clang that it should generate arm64 code anyways. There is
> no reason in trying to make ARM build with x86 instruction set.
> 
>> Also
>> this affects both toolstack builds and hypervisor. Is applying -march
>> like this actually appropriate for the toolstack?
> 
> This is a good question. I can't see why this can't be appropriate for
> toolstack. I.e. what bad can happen when building the toolstack.

In the future, we may want to build the tools for Armv8-M. So I think 
the -march should also applies for the toolstack.

Cheers,

-- 
Julien Grall

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Jan Beulich]

On 30.11.2024 18:15, Julien Grall wrote:
> On 29/11/2024 22:12, Volodymyr Babchuk wrote:
>> Jan Beulich <jbeulich@suse.com> writes:
>>> On 29.11.2024 02:49, Volodymyr Babchuk wrote:
>>>> --- a/config/arm64.mk
>>>> +++ b/config/arm64.mk
>>>> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>>>>   
>>>>   CFLAGS += #-marm -march= -mcpu= etc
>>>>   
>>>> +ifeq ($(clang),y)
>>>> +CFLAGS += -target aarch64 -march=armv8-a
>>>> +endif
>>>
>>> Why is this dependent on (just?) $(clang), not (also?) $(llvm)?
>>
>> Because this parameter is handled by clang only. There is no harm in
>> providing it explicitly. When building on arm64, value of this parameter
>> will match the default value for the platform. When building on x86, we
>> need to tell clang that it should generate arm64 code anyways. There is
>> no reason in trying to make ARM build with x86 instruction set.
>>
>>> Also
>>> this affects both toolstack builds and hypervisor. Is applying -march
>>> like this actually appropriate for the toolstack?
>>
>> This is a good question. I can't see why this can't be appropriate for
>> toolstack. I.e. what bad can happen when building the toolstack.
> 
> In the future, we may want to build the tools for Armv8-M. So I think 
> the -march should also applies for the toolstack.

Perhaps I don't know enough of the Arm world, but: Wouldn't it be possible
to build a tool stack suitable for a wide range for Arm64 flavors, while
requiring more targeted hypervisor binaries?

Jan

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Julien Grall]

Hi Jan,

On 02/12/2024 07:52, Jan Beulich wrote:
> On 30.11.2024 18:15, Julien Grall wrote:
>> On 29/11/2024 22:12, Volodymyr Babchuk wrote:
>>> Jan Beulich <jbeulich@suse.com> writes:
>>>> On 29.11.2024 02:49, Volodymyr Babchuk wrote:
>>>>> --- a/config/arm64.mk
>>>>> +++ b/config/arm64.mk
>>>>> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>>>>>    
>>>>>    CFLAGS += #-marm -march= -mcpu= etc
>>>>>    
>>>>> +ifeq ($(clang),y)
>>>>> +CFLAGS += -target aarch64 -march=armv8-a
>>>>> +endif
>>>>
>>>> Why is this dependent on (just?) $(clang), not (also?) $(llvm)?
>>>
>>> Because this parameter is handled by clang only. There is no harm in
>>> providing it explicitly. When building on arm64, value of this parameter
>>> will match the default value for the platform. When building on x86, we
>>> need to tell clang that it should generate arm64 code anyways. There is
>>> no reason in trying to make ARM build with x86 instruction set.
>>>
>>>> Also
>>>> this affects both toolstack builds and hypervisor. Is applying -march
>>>> like this actually appropriate for the toolstack?
>>>
>>> This is a good question. I can't see why this can't be appropriate for
>>> toolstack. I.e. what bad can happen when building the toolstack.
>>
>> In the future, we may want to build the tools for Armv8-M. So I think
>> the -march should also applies for the toolstack.
> 
> Perhaps I don't know enough of the Arm world, but: Wouldn't it be possible
> to build a tool stack suitable for a wide range for Arm64 flavors, while
> requiring more targeted hypervisor binaries?

Good question. There are some commonnality between ARMv8-M and ARMv8-R 
but I am not sure whether it means we could use -march=armv8-a build and 
run on ARMv8-M. Adding Ayan and Luca.

Cheers,

-- 
Julien Grall

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Luca Fancellu]

HI,

> On 2 Dec 2024, at 20:38, Julien Grall <julien@xen.org> wrote:
> 
> Hi Jan,
> 
> On 02/12/2024 07:52, Jan Beulich wrote:
>> On 30.11.2024 18:15, Julien Grall wrote:
>>> On 29/11/2024 22:12, Volodymyr Babchuk wrote:
>>>> Jan Beulich <jbeulich@suse.com> writes:
>>>>> On 29.11.2024 02:49, Volodymyr Babchuk wrote:
>>>>>> --- a/config/arm64.mk
>>>>>> +++ b/config/arm64.mk
>>>>>> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>>>>>>      CFLAGS += #-marm -march= -mcpu= etc
>>>>>>   +ifeq ($(clang),y)
>>>>>> +CFLAGS += -target aarch64 -march=armv8-a
>>>>>> +endif
>>>>> 
>>>>> Why is this dependent on (just?) $(clang), not (also?) $(llvm)?
>>>> 
>>>> Because this parameter is handled by clang only. There is no harm in
>>>> providing it explicitly. When building on arm64, value of this parameter
>>>> will match the default value for the platform. When building on x86, we
>>>> need to tell clang that it should generate arm64 code anyways. There is
>>>> no reason in trying to make ARM build with x86 instruction set.
>>>> 
>>>>> Also
>>>>> this affects both toolstack builds and hypervisor. Is applying -march
>>>>> like this actually appropriate for the toolstack?
>>>> 
>>>> This is a good question. I can't see why this can't be appropriate for
>>>> toolstack. I.e. what bad can happen when building the toolstack.
>>> 
>>> In the future, we may want to build the tools for Armv8-M. So I think
>>> the -march should also applies for the toolstack.
>> Perhaps I don't know enough of the Arm world, but: Wouldn't it be possible
>> to build a tool stack suitable for a wide range for Arm64 flavors, while
>> requiring more targeted hypervisor binaries?
> 
> Good question. There are some commonnality between ARMv8-M and ARMv8-R but I am not sure whether it means we could use -march=armv8-a build and run on ARMv8-M. Adding Ayan and Luca.

AFAIK a binary compiled for armv8-a aarch64 will work also on armv8-r aarch64 since they share almost everything apart from some MPU registers,
instead in order to work on armv8-m we might need to pass the appropriate march

Cheers,
Luca

Re: [PATCH 3/5] build: arm64: provide -target and -march if using clang

[Julien Grall]

Hi Volodymyr,

On 29/11/2024 01:49, Volodymyr Babchuk wrote:
> Provide -target and -march explicitly when building with clang. This
> makes cross-compilation much easier, because clang accept this
> parameters regardless of host platform. Basically,
> 
>    make XEN_TARGET_ARCH=arm64 clang=y llvm=y
> 
> will behave in the same way if building Xen on x86, or on arm64 or on
> any other platform.
> 
> -march is required because with default value, clang will not
> recognize EL2 registers.

Any chance this is happening because you are using "-target aarch64" 
rather than e.g. "arch64-arm-none-eabi"?

> Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
> ---
>   config/arm64.mk | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/config/arm64.mk b/config/arm64.mk
> index c4662f67d0..97eb9a82e7 100644
> --- a/config/arm64.mk
> +++ b/config/arm64.mk
> @@ -5,6 +5,10 @@ CONFIG_XEN_INSTALL_SUFFIX :=
>   
>   CFLAGS += #-marm -march= -mcpu= etc
>   
> +ifeq ($(clang),y)
> +CFLAGS += -target aarch64 -march=armv8-a

AFAIU, -target is the triplet similar to what one would set 
CROSS_COMPILE. If you don't specify some values, then it will assume the 
compiler defaults. I am not sure this is a good idea as this could 
change between compilers. So should we set the triplet properly (e.g. 
arch64-arm-none-eabi) or maybe let the user specify it via CROSS_COMPILE?

Regarding -march=armv8-a, if you want to keep it, then I think it should 
be outside of the 'if' because this could also apply for GCC.

Cheers,

-- 
Julien Grall

[PATCH 4/5] xen: build: add support for CFLAGS_REMOVE variable

[Volodymyr Babchuk]

This variable can be used in cases when we need to remove certain
CFLAGS for particular object file. One such case is
-mgeneral-regs-only flags that should be omitted when we are building
an object file that uses not only general registers.

Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
 xen/Rules.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/Rules.mk b/xen/Rules.mk
index d759cccee3..478318537f 100644
--- a/xen/Rules.mk
+++ b/xen/Rules.mk
@@ -179,7 +179,7 @@ cpp_flags = $(filter-out -Wa$(comma)% -flto,$(1))
 # Calculation of flags, first the generic flags, then the arch specific flags,
 # and last the flags modified for a target or a directory.
 
-c_flags = -MMD -MP -MF $(depfile) $(XEN_CFLAGS)
+c_flags = -MMD -MP -MF $(depfile) $(filter-out $(CFLAGS_REMOVE),$(XEN_CFLAGS))
 a_flags = -MMD -MP -MF $(depfile) $(XEN_AFLAGS)
 
 include $(srctree)/arch/$(SRCARCH)/Rules.mk
-- 
2.47.0

Re: [PATCH 4/5] xen: build: add support for CFLAGS_REMOVE variable

[Jan Beulich]

On 29.11.2024 02:49, Volodymyr Babchuk wrote:
> --- a/xen/Rules.mk
> +++ b/xen/Rules.mk
> @@ -179,7 +179,7 @@ cpp_flags = $(filter-out -Wa$(comma)% -flto,$(1))
>  # Calculation of flags, first the generic flags, then the arch specific flags,
>  # and last the flags modified for a target or a directory.
>  
> -c_flags = -MMD -MP -MF $(depfile) $(XEN_CFLAGS)
> +c_flags = -MMD -MP -MF $(depfile) $(filter-out $(CFLAGS_REMOVE),$(XEN_CFLAGS))

This looks rather fragile to me. Plus see the comment on the next patch,
where this is actually used.

Jan

[PATCH 5/5] xen: arm64: remove -mgeneral-regs-only for vfp.c

[Volodymyr Babchuk]

vfp.c actually accesses VFP registers, so it can't be built with
-mgeneral-regs-only flag when using clang, as clang will complain
about this:

arch/arm/arm64/vfp.c:9:18: error: instruction requires: fp-armv8
    9 |     asm volatile("stp q0, q1, [%1, #16 * 0]\n\t"
      |

Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
 xen/arch/arm/arm64/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile
index 6491c5350b..af949b8ee6 100644
--- a/xen/arch/arm/arm64/Makefile
+++ b/xen/arch/arm/arm64/Makefile
@@ -18,3 +18,5 @@ obj-$(CONFIG_ARM64_SVE) += sve.o sve-asm.o
 obj-y += traps.o
 obj-y += vfp.o
 obj-y += vsysreg.o
+
+$(obj)/vfp.o: CFLAGS_REMOVE += -mgeneral-regs-only
-- 
2.47.0

Re: [PATCH 5/5] xen: arm64: remove -mgeneral-regs-only for vfp.c

[Jan Beulich]

On 29.11.2024 02:49, Volodymyr Babchuk wrote:
> vfp.c actually accesses VFP registers, so it can't be built with
> -mgeneral-regs-only flag when using clang, as clang will complain
> about this:
> 
> arch/arm/arm64/vfp.c:9:18: error: instruction requires: fp-armv8
>     9 |     asm volatile("stp q0, q1, [%1, #16 * 0]\n\t"
>       |

Imo this needs addressing by inserting appropriate .arch_extension directives
in the asm(), to limit the scope of where VFP registers can be used to _just_
where we want them used.

> --- a/xen/arch/arm/arm64/Makefile
> +++ b/xen/arch/arm/arm64/Makefile
> @@ -18,3 +18,5 @@ obj-$(CONFIG_ARM64_SVE) += sve.o sve-asm.o
>  obj-y += traps.o
>  obj-y += vfp.o
>  obj-y += vsysreg.o
> +
> +$(obj)/vfp.o: CFLAGS_REMOVE += -mgeneral-regs-only

This, after all, allows the compiler to also use them behind our backs.

Jan