[PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

Using macros for inline assembly improves both readability and
compilation decisions that are distorted by big assembly blocks that use
alternative sections. Compile macros.S and use it to assemble all C
files. Currently, only x86 will use it.

Cc: Sam Ravnborg <sam@ravnborg.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: linux-kbuild@vger.kernel.org
Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Nadav Amit <namit@vmware.com>
---
 Makefile                 | 9 +++++++--
 arch/x86/Makefile        | 7 +++++++
 arch/x86/kernel/macros.S | 7 +++++++
 scripts/Kbuild.include   | 4 +++-
 scripts/mod/Makefile     | 2 ++
 5 files changed, 26 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/kernel/macros.S

diff --git a/Makefile b/Makefile
index 6c3da3e10f07..6c40d547513c 100644
--- a/Makefile
+++ b/Makefile
@@ -1071,7 +1071,7 @@ scripts: scripts_basic asm-generic gcc-plugins $(autoksyms_h)
 # version.h and scripts_basic is processed / created.
 
 # Listed in dependency order
-PHONY += prepare archprepare prepare0 prepare1 prepare2 prepare3
+PHONY += prepare archprepare macroprepare prepare0 prepare1 prepare2 prepare3
 
 # prepare3 is used to check if we are building in a separate output directory,
 # and if so do:
@@ -1094,7 +1094,9 @@ prepare2: prepare3 outputmakefile asm-generic
 prepare1: prepare2 $(version_h) $(autoksyms_h) include/generated/utsrelease.h
 	$(cmd_crmodverdir)
 
-archprepare: archheaders archscripts prepare1 scripts_basic
+macroprepare: prepare1 archmacros
+
+archprepare: archheaders archscripts macroprepare scripts_basic
 
 prepare0: archprepare gcc-plugins
 	$(Q)$(MAKE) $(build)=.
@@ -1162,6 +1164,9 @@ archheaders:
 PHONY += archscripts
 archscripts:
 
+PHONY += archmacros
+archmacros:
+
 PHONY += __headers
 __headers: $(version_h) scripts_basic uapi-asm-generic archheaders archscripts
 	$(Q)$(MAKE) $(build)=scripts build_unifdef
diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 8f6e7eb8ae9f..18a0084f134b 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -237,6 +237,13 @@ archscripts: scripts_basic
 archheaders:
 	$(Q)$(MAKE) $(build)=arch/x86/entry/syscalls all
 
+archmacros:
+	$(Q)$(MAKE) $(build)=arch/x86/kernel arch/x86/kernel/macros.s
+
+ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s -Wa,-
+export ASM_MACRO_FLAGS
+KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)
+
 ###
 # Kernel objects
 
diff --git a/arch/x86/kernel/macros.S b/arch/x86/kernel/macros.S
new file mode 100644
index 000000000000..cfc1c7d1a6eb
--- /dev/null
+++ b/arch/x86/kernel/macros.S
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file includes headers whose assembly part includes macros which are
+ * commonly used. The macros are precompiled into assmebly file which is later
+ * assembled together with each compiled file.
+ */
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index ce53639a864a..8aeb60eb6ee3 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -115,7 +115,9 @@ __cc-option = $(call try-run,\
 
 # Do not attempt to build with gcc plugins during cc-option tests.
 # (And this uses delayed resolution so the flags will be up to date.)
-CC_OPTION_CFLAGS = $(filter-out $(GCC_PLUGINS_CFLAGS),$(KBUILD_CFLAGS))
+# In addition, do not include the asm macros which are built later.
+CC_OPTION_FILTERED = $(GCC_PLUGINS_CFLAGS) $(ASM_MACRO_FLAGS)
+CC_OPTION_CFLAGS = $(filter-out $(CC_OPTION_FILTERED),$(KBUILD_CFLAGS))
 
 # cc-option
 # Usage: cflags-y += $(call cc-option,-march=winchip-c6,-march=i586)
diff --git a/scripts/mod/Makefile b/scripts/mod/Makefile
index 42c5d50f2bcc..a5b4af47987a 100644
--- a/scripts/mod/Makefile
+++ b/scripts/mod/Makefile
@@ -4,6 +4,8 @@ OBJECT_FILES_NON_STANDARD := y
 hostprogs-y	:= modpost mk_elfconfig
 always		:= $(hostprogs-y) empty.o
 
+CFLAGS_REMOVE_empty.o := $(ASM_MACRO_FLAGS)
+
 modpost-objs	:= modpost.o file2alias.o sumversion.o
 
 devicetable-offsets-file := devicetable-offsets.h
-- 
2.17.1

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 2018-10-03 3:30 p.m., Nadav Amit wrote:
> +ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s -Wa,-
> +export ASM_MACRO_FLAGS
> +KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)

I'm not sure how much we care about this or what we can do about it, but
adding the macros.s to the command line like this has effectively broken
distributed compiling with distcc and icecc as of v4.20-rc1. Neither
tool will successfully distribute any compile processes because the
non-local machines won't have a copy of macros.s.

Thanks,

Logan

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: Logan Gunthorpe
Sent: November 6, 2018 at 6:57:57 PM GMT
> To: Nadav Amit <namit@vmware.com>, Ingo Molnar <mingo@redhat.com>
> Cc: linux-kernel@vger.kernel.org>, x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, H. Peter Anvin <hpa@zytor.com>, linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> 
> 
> On 2018-10-03 3:30 p.m., Nadav Amit wrote:
>> +ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s -Wa,-
>> +export ASM_MACRO_FLAGS
>> +KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)
> 
> I'm not sure how much we care about this or what we can do about it, but
> adding the macros.s to the command line like this has effectively broken
> distributed compiling with distcc and icecc as of v4.20-rc1. Neither
> tool will successfully distribute any compile processes because the
> non-local machines won't have a copy of macros.s.

Err.. I don’t have a dist-cc environment. I wonder whether something like
that would do the trick:

-- >8 —

Subject: [PATCH] x86: Fix dist-cc

---
 arch/x86/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 5b562e464009..81d76cbcffff 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -237,9 +237,9 @@ archheaders:
 	$(Q)$(MAKE) $(build)=arch/x86/entry/syscalls all
 
 archmacros:
-	$(Q)$(MAKE) $(build)=arch/x86/kernel arch/x86/kernel/macros.s
+	$(Q)$(MAKE) $(build)=arch/x86/kernel ${objtree}/arch/x86/kernel/macros.s
 
-ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s -Wa,-
+ASM_MACRO_FLAGS = -Wa,${objtree}/arch/x86/kernel/macros.s -Wa,-
 export ASM_MACRO_FLAGS
 KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)
 
-- 
2.17.1

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 2018-11-06 12:18 p.m., Nadav Amit wrote:
> Err.. I don’t have a dist-cc environment. I wonder whether something like
> that would do the trick:

I tested it to be sure -- but, no, this does not have any affect on
distcc. Distcc usually does pre-processing locally and then sends the
single output of cpp to a remote for compilation and assembly. It
doesn't expect there to be additional assembly files tacked on the
command line and isn't going to handle that case correctly. Thus,
changing the path of the file won't fix anything.

Logan

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: Logan Gunthorpe
Sent: November 6, 2018 at 8:01:31 PM GMT
> To: Nadav Amit <namit@vmware.com>, Ingo Molnar <mingo@redhat.com>
> Cc: LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, H. Peter Anvin <hpa@zytor.com>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> 
> 
> On 2018-11-06 12:18 p.m., Nadav Amit wrote:
>> Err.. I don’t have a dist-cc environment. I wonder whether something like
>> that would do the trick:
> 
> I tested it to be sure -- but, no, this does not have any affect on
> distcc. Distcc usually does pre-processing locally and then sends the
> single output of cpp to a remote for compilation and assembly. It
> doesn't expect there to be additional assembly files tacked on the
> command line and isn't going to handle that case correctly. Thus,
> changing the path of the file won't fix anything.

Thanks for the explanation.

I played with dist-cc and gave it some thought, but I do not have a
solution. I thought that perhaps some hack is possible by using the same
preprocessed file can be used as both the source file and the assembly file,
but that’s as far as I managed to get.

Ideas? Do people care about it?

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 2018-11-07 11:01 a.m., Nadav Amit wrote:
> Ideas? Do people care about it?

Just spit balling, but is there a reason we didn't just put the macros
for inline assembly directly in the header? Something like this:

asm(".macro ANNOTATE_UNREACHABLE counter:req\n\t"
    "\\counter:\n\t"
        ".pushsection .discard.unreachable\n\t"
        ".long \\counter\\()b -.\n\t"
        ".popsection\n\t"
    ".endm\n");

#define annotate_unreachable() ({                   \
    asm volatile("ANNOTATE_UNREACHABLE counter=%c0" \
                 : : "i" (__COUNTER__));            \
})

It does mean the macros won't be usable in non-inline assembly, without
duplicating them, but do we care about that?

I would have expected people compiling the kernel with distcc to be
fairly common -- it certainly speeds up my kernel compiles quite
significantly and thus saves me a lot of time.

Logan

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: Logan Gunthorpe
Sent: November 7, 2018 at 6:53:02 PM GMT
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> 
> 
> On 2018-11-07 11:01 a.m., Nadav Amit wrote:
>> Ideas? Do people care about it?
> 
> Just spit balling, but is there a reason we didn't just put the macros
> for inline assembly directly in the header? Something like this:
> 
> asm(".macro ANNOTATE_UNREACHABLE counter:req\n\t"
>    "\\counter:\n\t"
>        ".pushsection .discard.unreachable\n\t"
>        ".long \\counter\\()b -.\n\t"
>        ".popsection\n\t"
>    ".endm\n");
> 
> #define annotate_unreachable() ({                   \
>    asm volatile("ANNOTATE_UNREACHABLE counter=%c0" \
>                 : : "i" (__COUNTER__));            \
> })
> 
> It does mean the macros won't be usable in non-inline assembly, without
> duplicating them, but do we care about that?

HPA indicated more than once that this is wrong (and that was indeed my
initial solution), since it is not guaranteed that the compiler would put
the macro assembly before its use.

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 2018-11-07 11:56 a.m., Nadav Amit wrote:
> HPA indicated more than once that this is wrong (and that was indeed my
> initial solution), since it is not guaranteed that the compiler would put
> the macro assembly before its use.

Hmm, that's very unfortunate. I don't really understand why the compiler
would not put the macro assembly in the same order as it appears in the
code and it would be in the correct order there.

In any case, I've submitted a couple of issues to icecc[1] and distcc[2]
to see if they have any ideas about supporting this on their sides.

Thanks,

Logan

[1] https://github.com/icecc/icecream/issues/428
[2] https://github.com/distcc/distcc/issues/312

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[hpa]

On November 7, 2018 1:43:39 PM PST, Logan Gunthorpe <logang@deltatee.com> wrote:
>
>
>On 2018-11-07 11:56 a.m., Nadav Amit wrote:
>> HPA indicated more than once that this is wrong (and that was indeed
>my
>> initial solution), since it is not guaranteed that the compiler would
>put
>> the macro assembly before its use.
>
>Hmm, that's very unfortunate. I don't really understand why the
>compiler
>would not put the macro assembly in the same order as it appears in the
>code and it would be in the correct order there.
>
>In any case, I've submitted a couple of issues to icecc[1] and
>distcc[2]
>to see if they have any ideas about supporting this on their sides.
>
>Thanks,
>
>Logan
>
>[1] https://github.com/icecc/icecream/issues/428
>[2] https://github.com/distcc/distcc/issues/312

Apparently gcc will treat them like basic blocks and possibly move them around.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: hpa@zytor.com
Sent: November 7, 2018 at 9:50:28 PM GMT
> To: Logan Gunthorpe <logang@deltatee.com>, Nadav Amit <namit@vmware.com>, Ingo Molnar <mingo@redhat.com>
> Cc: LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> On November 7, 2018 1:43:39 PM PST, Logan Gunthorpe <logang@deltatee.com> wrote:
>> On 2018-11-07 11:56 a.m., Nadav Amit wrote:
>>> HPA indicated more than once that this is wrong (and that was indeed
>> my
>>> initial solution), since it is not guaranteed that the compiler would
>> put
>>> the macro assembly before its use.
>> 
>> Hmm, that's very unfortunate. I don't really understand why the
>> compiler
>> would not put the macro assembly in the same order as it appears in the
>> code and it would be in the correct order there.
>> 
>> In any case, I've submitted a couple of issues to icecc[1] and
>> distcc[2]
>> to see if they have any ideas about supporting this on their sides.
>> 
>> Thanks,
>> 
>> Logan
>> 
>> [1] https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ficecc%2Ficecream%2Fissues%2F428&amp;data=02%7C01%7Cnamit%40vmware.com%7C30ab3751343b49f869ab08d644fb1d8c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636772242666772528&amp;sdata=dXKTR79LkFDQ9IXxYw9XYt0VPFa4MXrMUcc86w5uy%2Fk%3D&amp;reserved=0
>> [2] https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdistcc%2Fdistcc%2Fissues%2F312&amp;data=02%7C01%7Cnamit%40vmware.com%7C30ab3751343b49f869ab08d644fb1d8c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636772242666772528&amp;sdata=XynZ1bFbKAb8V2eoPQbXudEJ%2B%2Bu8QA3mM4Sr4E%2FTzWs%3D&amp;reserved=0
> 
> Apparently gcc will treat them like basic blocks and possibly move them around.

Maybe it is possible to break the compilation of each object into two
stages: first, compile the source without assembly, and then take the
generated .s file and assemble it with the .s file of the macros.

Does it sounds as something that may work? I guess it should only be done
when distcc is used.

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 2018-11-07 11:18 p.m., Nadav Amit wrote:
>> Apparently gcc will treat them like basic blocks and possibly move them around.
> 
> Maybe it is possible to break the compilation of each object into two
> stages: first, compile the source without assembly, and then take the
> generated .s file and assemble it with the .s file of the macros.
> 
> Does it sounds as something that may work? I guess it should only be done
> when distcc is used.

In theory it would at least allow the compile step to be distributed,
the assembly step would still have to be done locally... It'd be better
than nothing, I guess.

It'd also be difficult to know when distribution is being done and that
it's necessary to split the steps. We'd have to add an environment
variable or something and users would need to know they have to set it
when using a distributed compile.

Logan

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: Logan Gunthorpe
Sent: November 8, 2018 at 5:14:58 PM GMT
> To: Nadav Amit <namit@vmware.com>, hpa@zytor.com <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>
> Cc: LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> 
> 
> On 2018-11-07 11:18 p.m., Nadav Amit wrote:
>>> Apparently gcc will treat them like basic blocks and possibly move them around.
>> 
>> Maybe it is possible to break the compilation of each object into two
>> stages: first, compile the source without assembly, and then take the
>> generated .s file and assemble it with the .s file of the macros.
>> 
>> Does it sounds as something that may work? I guess it should only be done
>> when distcc is used.
> 
> In theory it would at least allow the compile step to be distributed,
> the assembly step would still have to be done locally... It'd be better
> than nothing, I guess.

I don’t think the assembly stage needs to be done locally. gcc can still be
used to deploy the assembler. I am not too familiar with distcc, so I don’t
know whether the preprocessing supports multiple source-files, and whether
it has some strange-behavior when it comes to .S/.s files.

Well, I’ll give it a try.

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 2018-11-08 12:54 p.m., Nadav Amit wrote:
> I don’t think the assembly stage needs to be done locally. gcc can still be
> used to deploy the assembler. I am not too familiar with distcc, so I don’t
> know whether the preprocessing supports multiple source-files, and whether
> it has some strange-behavior when it comes to .S/.s files.

The problem is that it's the assembly stage that needs the extra
macros.s and without that file being transferred somehow, that stage
must be done locally.

The distcc guys have a thought that the macros.s file might be able to
be transferred using pump mode with a change to include_server.py. I
don't think that will work for icecc though.

Logan

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: Logan Gunthorpe
Sent: November 8, 2018 at 8:00:33 PM GMT
> To: Nadav Amit <namit@vmware.com>, hpa@zytor.com <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>
> Cc: LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> 
> 
> On 2018-11-08 12:54 p.m., Nadav Amit wrote:
>> I don’t think the assembly stage needs to be done locally. gcc can still be
>> used to deploy the assembler. I am not too familiar with distcc, so I don’t
>> know whether the preprocessing supports multiple source-files, and whether
>> it has some strange-behavior when it comes to .S/.s files.
> 
> The problem is that it's the assembly stage that needs the extra
> macros.s and without that file being transferred somehow, that stage
> must be done locally.

I understand, but the idea is to have two stages, for instance:

  gcc ... -S -o .tmp_[unit].S [unit].c

and then

  gcc ... -D__ASSEMBLY__ arch/x86/kernel/macros.S .tmp_[unit].S

(Yes, I realize the .tmp_[unit].S was already preprocessed, but this way you
can combine both inputs)

Unfortunately, as I write this email (and run tests) I realize distcc is too
dumb to handle two input files:

  "(dcc_scan_args) do we have two inputs?  i give up "

Just great. So I guess macros.s would need to be concatenated with
.tmp_[unit].s as a separate (local) interim stage.

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Nadav Amit]

From: Nadav Amit
Sent: November 8, 2018 at 8:18:23 PM GMT
> To: Logan Gunthorpe <logang@deltatee.com>, hpa@zytor.com <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>
> Cc: LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
> 
> 
> From: Logan Gunthorpe
> Sent: November 8, 2018 at 8:00:33 PM GMT
>> To: Nadav Amit <namit@vmware.com>, hpa@zytor.com <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>
>> Cc: LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>, Sam Ravnborg <sam@ravnborg.org>, Michal Marek <michal.lkml@markovi.net>, Thomas Gleixner <tglx@linutronix.de>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Stephen Bates <sbates@raithlin.com>
>> Subject: Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm
>> 
>> 
>> 
>> 
>> On 2018-11-08 12:54 p.m., Nadav Amit wrote:
>>> I don’t think the assembly stage needs to be done locally. gcc can still be
>>> used to deploy the assembler. I am not too familiar with distcc, so I don’t
>>> know whether the preprocessing supports multiple source-files, and whether
>>> it has some strange-behavior when it comes to .S/.s files.
>> 
>> The problem is that it's the assembly stage that needs the extra
>> macros.s and without that file being transferred somehow, that stage
>> must be done locally.
> 
> I understand, but the idea is to have two stages, for instance:
> 
>  gcc ... -S -o .tmp_[unit].S [unit].c
> 
> and then
> 
>  gcc ... -D__ASSEMBLY__ arch/x86/kernel/macros.S .tmp_[unit].S
> 
> (Yes, I realize the .tmp_[unit].S was already preprocessed, but this way you
> can combine both inputs)
> 
> Unfortunately, as I write this email (and run tests) I realize distcc is too
> dumb to handle two input files:
> 
>  "(dcc_scan_args) do we have two inputs?  i give up "
> 
> Just great. So I guess macros.s would need to be concatenated with
> .tmp_[unit].s as a separate (local) interim stage.

Err.. I hate makefiles and distcc doesn’t make life easier. For instance, if
it sees two source files on the command-line, it freaks out and compiles it
locally. It also has an option to distribute assembly, but it is not enabled
by default.

Anyhow, can you try the following patch?

-- >8 --

Subject: [PATCH] Makefile: Fix distcc compilation with x86 macros

Introducing the use of asm macros in c-code broke distcc, since it only
sends the preprocessed source file. The solution is to break the
compilation into two separate phases of compilation and assembly, and
between the two concatanate the assembly macros and the compiled (yet
not assembled) source file. Since this is less efficient, this
compilation mode is only used when make is called with the "DISTCC=y"
parameter.

Note that the assembly stage should also be distributable, if distcc is
configured using "CFLAGS=-DENABLE_REMOTE_ASSEMBLE".

Reported-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Nadav Amit <namit@vmware.com>
---
 Makefile               |  4 +++-
 arch/x86/Makefile      |  4 +++-
 scripts/Makefile.build | 29 +++++++++++++++++++++++++++--
 3 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 9fce8b91c15f..c07349fc38c7 100644
--- a/Makefile
+++ b/Makefile
@@ -743,7 +743,9 @@ KBUILD_CFLAGS   += $(call cc-option, -gsplit-dwarf, -g)
 else
 KBUILD_CFLAGS	+= -g
 endif
-KBUILD_AFLAGS	+= -Wa,-gdwarf-2
+AFLAGS_DEBUG_INFO = -Wa,-gdwarf-2
+export AFLAGS_DEBUG_INFO
+KBUILD_AFLAGS	+= $(AFLAGS_DEBUG_INFO)
 endif
 ifdef CONFIG_DEBUG_INFO_DWARF4
 KBUILD_CFLAGS	+= $(call cc-option, -gdwarf-4,)
diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index f5d7f4134524..080bd9cbc4e1 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -238,7 +238,9 @@ archheaders:
 archmacros:
 	$(Q)$(MAKE) $(build)=arch/x86/kernel arch/x86/kernel/macros.s
 
-ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s
+ASM_MACRO_FILE = arch/x86/kernel/macros.s
+export ASM_MACRO_FILE
+ASM_MACRO_FLAGS = -Wa,$(ASM_MACRO_FILE)
 export ASM_MACRO_FLAGS
 KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)
 
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 6a6be9f440cf..2b79789a3e10 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -155,8 +155,33 @@ $(obj)/%.ll: $(src)/%.c FORCE
 
 quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
 
+# If distcc is used, then when an assembly macro files is needed, the
+# compilation stage and the assembly stage need to be separated. Providing
+# "DISTCC=y" option enables the separate compilation and assembly.
+cmd_cc_o_c_simple = $(CC) $(c_flags) -c -o $(1) $<
+
+ifeq ($(DISTCC),y)
+a_flags_no_debug = $(filter-out $(AFLAGS_DEBUG_INFO), $(a_flags))
+c_flags_no_macros = $(filter-out $(ASM_MACRO_FLAGS), $(c_flags))
+
+cmd_cc_o_c_two_steps =							\
+	$(CC) $(c_flags_no_macros) $(DISABLE_LTO) -fverbose-asm -S 	\
+		-o $(@D)/.$(@F:.o=.s) $< ;				\
+	cat $(ASM_MACRO_FILE) $(@D)/.$(@F:.o=.s) > 			\
+		$(@D)/.tmp_$(@F:.o=.s);					\
+	$(CC) $(a_flags_no_debug) -c -o $(1) $(@D)/.tmp_$(@F:.o=.s) ; 	\
+	rm -f $(@D)/.$(@F:.o=.s) $(@D)/.tmp_$(@F:.o=.s)			\
+
+cmd_cc_o_c_helper =    							\
+	$(if $(findstring $(ASM_MACRO_FLAGS),$(c_flags)),		\
+		$(call cmd_cc_o_c_two_steps, $(1)),			\
+		$(call cmd_cc_o_c_simple, $(1)))
+else
+cmd_cc_o_c_helper = $(call cmd_cc_o_c_simple, $(1))
+endif
+
 ifndef CONFIG_MODVERSIONS
-cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $<
+cmd_cc_o_c = $(call cmd_cc_o_c_helper,$@)
 
 else
 # When module versioning is enabled the following steps are executed:
@@ -171,7 +196,7 @@ else
 #   replace the unresolved symbols __crc_exported_symbol with
 #   the actual value of the checksum generated by genksyms
 
-cmd_cc_o_c = $(CC) $(c_flags) -c -o $(@D)/.tmp_$(@F) $<
+cmd_cc_o_c = $(call cmd_cc_o_c_helper,$(@D)/.tmp_$(@F))
 
 cmd_modversions_c =								\
 	if $(OBJDUMP) -h $(@D)/.tmp_$(@F) | grep -q __ksymtab; then		\
-- 
2.17.1

Re: [PATCH v9 02/10] Makefile: Prepare for using macros for inline asm

[Logan Gunthorpe]

On 10/11/18 03:04 PM, Nadav Amit wrote:
> Err.. I hate makefiles and distcc doesn’t make life easier. For instance, if
> it sees two source files on the command-line, it freaks out and compiles it
> locally. It also has an option to distribute assembly, but it is not enabled
> by default.
> 
> Anyhow, can you try the following patch?

Very nice, great work! It works as expected. I tested with both distcc
and icecc and both work well with DISTCC=y.

Also worth noting, is that distcc has a patch to work without DISTCC=y
in pump mode (though pump mode doesn't work super great with the kernel
as some of the include processing takes a long time). When I have some
free time, I may also try to patch icecc to work with it, as in my
experience, it performs better.

Regardless, I think your patch would be very valuable to be in upstream
so there's at least some way to use the existing versions of distcc and
icecc.

Thanks!

Logan