* [PATCH 2/3] kbuild: do not create intermediate *.tar for source tarballs
2023-04-07 10:16 [PATCH 1/3] kbuild: merge cmd_archive_linux and cmd_archive_perf Masahiro Yamada
@ 2023-04-07 10:16 ` Masahiro Yamada
2023-04-07 18:11 ` Nathan Chancellor
2023-04-07 10:16 ` [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages Masahiro Yamada
2023-04-07 18:00 ` [PATCH 1/3] kbuild: merge cmd_archive_linux and cmd_archive_perf Nathan Chancellor
2 siblings, 1 reply; 12+ messages in thread
From: Masahiro Yamada @ 2023-04-07 10:16 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-kernel, Masahiro Yamada, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier
Since commit 05e96e96a315 ("kbuild: use git-archive for source package
creation"), source tarballs are created in two steps; create *.tar file
then compress it. I split the compression as a separate rule because I
just thought 'git archive' supported only gzip for compression. I admit
the unneeded *.tar file is annoying.
For other compression algorithms, I could pipe the two commands:
$ git archive HEAD | xz > linux.tar.xz
I read git-archive(1) carefully, and I realized GIT had provided a
more elegant way:
$ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD
This commit uses 'tar.tar.*.command' configuration to specify the
compression backend so we can create a compressed tarball directly.
GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
is more than a decade old, so it should be available on almost all build
environments.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
scripts/Makefile.package | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index a205617730c6..7707975f729b 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -57,16 +57,23 @@ check-git:
false; \
fi
+archive-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)"
+archive-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)"
+archive-config-tar.xz = -c tar.tar.xz.command="$(XZ)"
+archive-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)"
+
quiet_cmd_archive = ARCHIVE $@
- cmd_archive = git -C $(srctree) archive \
+ cmd_archive = git -C $(srctree) $(archive-config-tar$(suffix $@)) archive \
--output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
# Linux source tarball
# ---------------------------------------------------------------------------
-targets += linux.tar
-linux.tar: archive-args = $$(cat $<)
-linux.tar: .tmp_HEAD FORCE
+linux-tarballs := $(addprefix linux, .tar.gz)
+
+targets += $(linux-tarballs)
+$(linux-tarballs): archive-args = $$(cat $<)
+$(linux-tarballs): .tmp_HEAD FORCE
$(call if_changed,archive)
# rpm-pkg
@@ -185,9 +192,12 @@ perf-archive-args = --add-file=$$(realpath $(word 2, $^)) \
--add-file=$$(realpath $(word 3, $^)) \
$$(cat $(word 2, $^))^{tree} $$(cat $<)
-targets += perf-$(KERNELVERSION).tar
-perf-$(KERNELVERSION).tar: archive-args = $(perf-archive-args)
-perf-$(KERNELVERSION).tar: tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
+
+perf-tarballs := $(addprefix perf-$(KERNELVERSION), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst)
+
+targets += $(perf-tarballs)
+$(perf-tarballs): archive-args = $(perf-archive-args)
+$(perf-tarballs): tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
$(call if_changed,archive)
PHONY += perf-tar-src-pkg
--
2.37.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 2/3] kbuild: do not create intermediate *.tar for source tarballs
2023-04-07 10:16 ` [PATCH 2/3] kbuild: do not create intermediate *.tar for source tarballs Masahiro Yamada
@ 2023-04-07 18:11 ` Nathan Chancellor
2023-04-08 1:32 ` Masahiro Yamada
0 siblings, 1 reply; 12+ messages in thread
From: Nathan Chancellor @ 2023-04-07 18:11 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Nick Desaulniers, Nicolas Schier
On Fri, Apr 07, 2023 at 07:16:28PM +0900, Masahiro Yamada wrote:
> Since commit 05e96e96a315 ("kbuild: use git-archive for source package
> creation"), source tarballs are created in two steps; create *.tar file
> then compress it. I split the compression as a separate rule because I
> just thought 'git archive' supported only gzip for compression. I admit
> the unneeded *.tar file is annoying.
>
> For other compression algorithms, I could pipe the two commands:
>
> $ git archive HEAD | xz > linux.tar.xz
>
> I read git-archive(1) carefully, and I realized GIT had provided a
> more elegant way:
Hooray for documentation :)
> $ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD
>
> This commit uses 'tar.tar.*.command' configuration to specify the
> compression backend so we can create a compressed tarball directly.
>
> GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
> is more than a decade old, so it should be available on almost all build
> environments.
git 1.7.7 it seems, certainly ancientware in my opinion. If people have
issues with this, they can just upgrade git; even RHEL7 has git 1.8.x.
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> ---
>
> scripts/Makefile.package | 24 +++++++++++++++++-------
> 1 file changed, 17 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> index a205617730c6..7707975f729b 100644
> --- a/scripts/Makefile.package
> +++ b/scripts/Makefile.package
> @@ -57,16 +57,23 @@ check-git:
> false; \
> fi
>
> +archive-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)"
> +archive-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)"
> +archive-config-tar.xz = -c tar.tar.xz.command="$(XZ)"
> +archive-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)"
> +
> quiet_cmd_archive = ARCHIVE $@
> - cmd_archive = git -C $(srctree) archive \
> + cmd_archive = git -C $(srctree) $(archive-config-tar$(suffix $@)) archive \
> --output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
>
> # Linux source tarball
> # ---------------------------------------------------------------------------
>
> -targets += linux.tar
> -linux.tar: archive-args = $$(cat $<)
> -linux.tar: .tmp_HEAD FORCE
> +linux-tarballs := $(addprefix linux, .tar.gz)
Is there any reason not to allow other compression formats for linux
like you do for perf?
> +
> +targets += $(linux-tarballs)
> +$(linux-tarballs): archive-args = $$(cat $<)
> +$(linux-tarballs): .tmp_HEAD FORCE
> $(call if_changed,archive)
>
> # rpm-pkg
> @@ -185,9 +192,12 @@ perf-archive-args = --add-file=$$(realpath $(word 2, $^)) \
> --add-file=$$(realpath $(word 3, $^)) \
> $$(cat $(word 2, $^))^{tree} $$(cat $<)
>
> -targets += perf-$(KERNELVERSION).tar
> -perf-$(KERNELVERSION).tar: archive-args = $(perf-archive-args)
> -perf-$(KERNELVERSION).tar: tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
> +
> +perf-tarballs := $(addprefix perf-$(KERNELVERSION), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst)
> +
> +targets += $(perf-tarballs)
> +$(perf-tarballs): archive-args = $(perf-archive-args)
> +$(perf-tarballs): tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
> $(call if_changed,archive)
>
> PHONY += perf-tar-src-pkg
> --
> 2.37.2
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH 2/3] kbuild: do not create intermediate *.tar for source tarballs
2023-04-07 18:11 ` Nathan Chancellor
@ 2023-04-08 1:32 ` Masahiro Yamada
0 siblings, 0 replies; 12+ messages in thread
From: Masahiro Yamada @ 2023-04-08 1:32 UTC (permalink / raw)
To: Nathan Chancellor
Cc: linux-kbuild, linux-kernel, Nick Desaulniers, Nicolas Schier
On Sat, Apr 8, 2023 at 3:11 AM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Fri, Apr 07, 2023 at 07:16:28PM +0900, Masahiro Yamada wrote:
> > Since commit 05e96e96a315 ("kbuild: use git-archive for source package
> > creation"), source tarballs are created in two steps; create *.tar file
> > then compress it. I split the compression as a separate rule because I
> > just thought 'git archive' supported only gzip for compression. I admit
> > the unneeded *.tar file is annoying.
> >
> > For other compression algorithms, I could pipe the two commands:
> >
> > $ git archive HEAD | xz > linux.tar.xz
> >
> > I read git-archive(1) carefully, and I realized GIT had provided a
> > more elegant way:
>
> Hooray for documentation :)
>
> > $ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD
> >
> > This commit uses 'tar.tar.*.command' configuration to specify the
> > compression backend so we can create a compressed tarball directly.
> >
> > GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
> > is more than a decade old, so it should be available on almost all build
> > environments.
>
> git 1.7.7 it seems, certainly ancientware in my opinion. If people have
> issues with this, they can just upgrade git; even RHEL7 has git 1.8.x.
>
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
>
> Reviewed-by: Nathan Chancellor <nathan@kernel.org>
>
> > ---
> >
> > scripts/Makefile.package | 24 +++++++++++++++++-------
> > 1 file changed, 17 insertions(+), 7 deletions(-)
> >
> > diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> > index a205617730c6..7707975f729b 100644
> > --- a/scripts/Makefile.package
> > +++ b/scripts/Makefile.package
> > @@ -57,16 +57,23 @@ check-git:
> > false; \
> > fi
> >
> > +archive-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)"
> > +archive-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)"
> > +archive-config-tar.xz = -c tar.tar.xz.command="$(XZ)"
> > +archive-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)"
> > +
> > quiet_cmd_archive = ARCHIVE $@
> > - cmd_archive = git -C $(srctree) archive \
> > + cmd_archive = git -C $(srctree) $(archive-config-tar$(suffix $@)) archive \
> > --output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
> >
> > # Linux source tarball
> > # ---------------------------------------------------------------------------
> >
> > -targets += linux.tar
> > -linux.tar: archive-args = $$(cat $<)
> > -linux.tar: .tmp_HEAD FORCE
> > +linux-tarballs := $(addprefix linux, .tar.gz)
>
> Is there any reason not to allow other compression formats for linux
> like you do for perf?
Currently, gzip is only allowed because there is no way to specify
the compression algorithm.
I already have a patch locally, but not submitted yet.
I prioritize groundwork, then add new features later.
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-07 10:16 [PATCH 1/3] kbuild: merge cmd_archive_linux and cmd_archive_perf Masahiro Yamada
2023-04-07 10:16 ` [PATCH 2/3] kbuild: do not create intermediate *.tar for source tarballs Masahiro Yamada
@ 2023-04-07 10:16 ` Masahiro Yamada
2023-04-07 18:12 ` Nathan Chancellor
2023-04-11 0:33 ` Masahiro Yamada
2023-04-07 18:00 ` [PATCH 1/3] kbuild: merge cmd_archive_linux and cmd_archive_perf Nathan Chancellor
2 siblings, 2 replies; 12+ messages in thread
From: Masahiro Yamada @ 2023-04-07 10:16 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-kernel, Masahiro Yamada, Nathan Chancellor,
Nick Desaulniers, Nick Terrell, Nicolas Schier
Commit 05e96e96a315 ("kbuild: use git-archive for source package
creation") split the compression as a separate step to factor out
the common build rules.
With the previous commit, we got back to the situation where
compressed source tarballs are created by a single rule.
There is no reason to keep the separate compression rules.
Generate the comressed tar packages directly.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
scripts/Makefile.package | 27 +++++++++------------------
1 file changed, 9 insertions(+), 18 deletions(-)
diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 7707975f729b..e0e18d7dfbd5 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -2,7 +2,6 @@
# Makefile for the different targets used to generate full packages of a kernel
include $(srctree)/scripts/Kbuild.include
-include $(srctree)/scripts/Makefile.lib
KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
KBUILD_PKG_ROOTCMD ?="fakeroot -u"
@@ -27,21 +26,6 @@ fi ; \
tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
-# tarball compression
-# ---------------------------------------------------------------------------
-
-%.tar.gz: %.tar
- $(call cmd,gzip)
-
-%.tar.bz2: %.tar
- $(call cmd,bzip2)
-
-%.tar.xz: %.tar
- $(call cmd,xzmisc)
-
-%.tar.zst: %.tar
- $(call cmd,zstd)
-
# Git
# ---------------------------------------------------------------------------
@@ -153,10 +137,17 @@ tar-install: FORCE
$(Q)$(MAKE) -f $(srctree)/Makefile
+$(Q)$(srctree)/scripts/package/buildtar $@
+compress-tar.gz = -I "$(KGZIP)"
+compress-tar.bz2 = -I "$(KBZIP2)"
+compress-tar.xz = -I "$(XZ)"
+compress-tar.zst = -I "$(ZSTD)"
+
quiet_cmd_tar = TAR $@
- cmd_tar = cd $<; tar cf ../$@ --owner=root --group=root --sort=name *
+ cmd_tar = cd $<; tar cf ../$@ $(compress-tar$(suffix $@)) --owner=root --group=root --sort=name *
+
+dir-tarballs := $(addprefix linux-$(KERNELRELEASE)-$(ARCH), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst)
-linux-$(KERNELRELEASE)-$(ARCH).tar: tar-install
+$(dir-tarballs): tar-install
$(call cmd,tar)
PHONY += dir-pkg
--
2.37.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-07 10:16 ` [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages Masahiro Yamada
@ 2023-04-07 18:12 ` Nathan Chancellor
2023-04-20 8:54 ` Tariq Toukan
2023-04-11 0:33 ` Masahiro Yamada
1 sibling, 1 reply; 12+ messages in thread
From: Nathan Chancellor @ 2023-04-07 18:12 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Nick Desaulniers, Nick Terrell,
Nicolas Schier
On Fri, Apr 07, 2023 at 07:16:29PM +0900, Masahiro Yamada wrote:
> Commit 05e96e96a315 ("kbuild: use git-archive for source package
> creation") split the compression as a separate step to factor out
> the common build rules.
>
> With the previous commit, we got back to the situation where
> compressed source tarballs are created by a single rule.
> There is no reason to keep the separate compression rules.
>
> Generate the comressed tar packages directly.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> ---
>
> scripts/Makefile.package | 27 +++++++++------------------
> 1 file changed, 9 insertions(+), 18 deletions(-)
>
> diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> index 7707975f729b..e0e18d7dfbd5 100644
> --- a/scripts/Makefile.package
> +++ b/scripts/Makefile.package
> @@ -2,7 +2,6 @@
> # Makefile for the different targets used to generate full packages of a kernel
>
> include $(srctree)/scripts/Kbuild.include
> -include $(srctree)/scripts/Makefile.lib
>
> KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE))
> KBUILD_PKG_ROOTCMD ?="fakeroot -u"
> @@ -27,21 +26,6 @@ fi ; \
> tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
> --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3)
>
> -# tarball compression
> -# ---------------------------------------------------------------------------
> -
> -%.tar.gz: %.tar
> - $(call cmd,gzip)
> -
> -%.tar.bz2: %.tar
> - $(call cmd,bzip2)
> -
> -%.tar.xz: %.tar
> - $(call cmd,xzmisc)
> -
> -%.tar.zst: %.tar
> - $(call cmd,zstd)
> -
> # Git
> # ---------------------------------------------------------------------------
>
> @@ -153,10 +137,17 @@ tar-install: FORCE
> $(Q)$(MAKE) -f $(srctree)/Makefile
> +$(Q)$(srctree)/scripts/package/buildtar $@
>
> +compress-tar.gz = -I "$(KGZIP)"
> +compress-tar.bz2 = -I "$(KBZIP2)"
> +compress-tar.xz = -I "$(XZ)"
> +compress-tar.zst = -I "$(ZSTD)"
> +
> quiet_cmd_tar = TAR $@
> - cmd_tar = cd $<; tar cf ../$@ --owner=root --group=root --sort=name *
> + cmd_tar = cd $<; tar cf ../$@ $(compress-tar$(suffix $@)) --owner=root --group=root --sort=name *
> +
> +dir-tarballs := $(addprefix linux-$(KERNELRELEASE)-$(ARCH), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst)
>
> -linux-$(KERNELRELEASE)-$(ARCH).tar: tar-install
> +$(dir-tarballs): tar-install
> $(call cmd,tar)
>
> PHONY += dir-pkg
> --
> 2.37.2
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-07 18:12 ` Nathan Chancellor
@ 2023-04-20 8:54 ` Tariq Toukan
2023-04-20 9:06 ` Nicolas Schier
0 siblings, 1 reply; 12+ messages in thread
From: Tariq Toukan @ 2023-04-20 8:54 UTC (permalink / raw)
To: Nathan Chancellor, Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Nick Desaulniers, Nick Terrell,
Nicolas Schier, dalevi, Gal Pressman, Leon Romanovsky,
Saeed Mahameed, Tariq Toukan
On 07/04/2023 21:12, Nathan Chancellor wrote:
> On Fri, Apr 07, 2023 at 07:16:29PM +0900, Masahiro Yamada wrote:
>> Commit 05e96e96a315 ("kbuild: use git-archive for source package
>> creation") split the compression as a separate step to factor out
>> the common build rules.
>>
>> With the previous commit, we got back to the situation where
>> compressed source tarballs are created by a single rule.
>> There is no reason to keep the separate compression rules.
>>
>> Generate the comressed tar packages directly.
>>
>> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
>
> Reviewed-by: Nathan Chancellor <nathan@kernel.org>
>
Hi,
We started seeing the failure below in rc7.
We narrowed it down to your patches:
3c65a2704cdd kbuild: do not create intermediate *.tar for tar packages
f8d94++c4e403c kbuild: do not create intermediate *.tar for source tarballs
f6d8283549bc kbuild: merge cmd_archive_linux and cmd_archive_perf
aa7d233f45b4 kbuild: give up untracked files for source package builds
Can you please take a look and advise?
Regards,
Tariq
[root@c-237-113-200-203 linux]# make -j24 rpm-pkg
sh ./scripts/package/mkspec >./kernel.spec
rpmbuild --target x86_64-linux -bs kernel.spec \
--define='_smp_mflags %{nil}' --define='_sourcedir rpmbuild/SOURCES'
--define='_srcrpmdir .'
Building target platforms: x86_64-linux
Building for target x86_64-linux
Wrote: ./kernel-6.3.0_rc7+-1.src.rpm
rpmbuild --target x86_64-linux -rb kernel-6.3.0_rc7+-1.src.rpm \
--define='_smp_mflags %{nil}'
Installing kernel-6.3.0_rc7+-1.src.rpm
Building target platforms: x86_64-linux
Building for target x86_64-linux
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.yDFEga
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd /root/rpmbuild/BUILD
+ rm -rf linux
+ /usr/bin/gzip -dc /root/rpmbuild/SOURCES/linux.tar.gz
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd linux
/var/tmp/rpm-tmp.yDFEga: line 37: cd: linux: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.yDFEga (%prep)
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.yDFEga (%prep)
make[1]: *** [scripts/Makefile.package:69: rpm-pkg] Error 1
make: *** [Makefile:1656: rpm-pkg] Error 2
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-20 8:54 ` Tariq Toukan
@ 2023-04-20 9:06 ` Nicolas Schier
2023-04-20 9:19 ` Leon Romanovsky
0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Schier @ 2023-04-20 9:06 UTC (permalink / raw)
To: Tariq Toukan
Cc: Nathan Chancellor, Masahiro Yamada, linux-kbuild, linux-kernel,
Nick Desaulniers, Nick Terrell, dalevi, Gal Pressman,
Leon Romanovsky, Saeed Mahameed, Tariq Toukan
On Thu, Apr 20, 2023 at 11:54:34AM +0300, Tariq Toukan wrote:
>
>
> On 07/04/2023 21:12, Nathan Chancellor wrote:
> > On Fri, Apr 07, 2023 at 07:16:29PM +0900, Masahiro Yamada wrote:
> > > Commit 05e96e96a315 ("kbuild: use git-archive for source package
> > > creation") split the compression as a separate step to factor out
> > > the common build rules.
> > >
> > > With the previous commit, we got back to the situation where
> > > compressed source tarballs are created by a single rule.
> > > There is no reason to keep the separate compression rules.
> > >
> > > Generate the comressed tar packages directly.
> > >
> > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> >
> > Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> >
>
> Hi,
>
> We started seeing the failure below in rc7.
> We narrowed it down to your patches:
>
> 3c65a2704cdd kbuild: do not create intermediate *.tar for tar packages
> f8d94++c4e403c kbuild: do not create intermediate *.tar for source tarballs
> f6d8283549bc kbuild: merge cmd_archive_linux and cmd_archive_perf
> aa7d233f45b4 kbuild: give up untracked files for source package builds
>
> Can you please take a look and advise?
>
> Regards,
> Tariq
>
> [root@c-237-113-200-203 linux]# make -j24 rpm-pkg
> sh ./scripts/package/mkspec >./kernel.spec
> rpmbuild --target x86_64-linux -bs kernel.spec \
> --define='_smp_mflags %{nil}' --define='_sourcedir rpmbuild/SOURCES'
> --define='_srcrpmdir .'
> Building target platforms: x86_64-linux
> Building for target x86_64-linux
> Wrote: ./kernel-6.3.0_rc7+-1.src.rpm
> rpmbuild --target x86_64-linux -rb kernel-6.3.0_rc7+-1.src.rpm \
> --define='_smp_mflags %{nil}'
> Installing kernel-6.3.0_rc7+-1.src.rpm
> Building target platforms: x86_64-linux
> Building for target x86_64-linux
> Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.yDFEga
> + umask 022
> + cd /root/rpmbuild/BUILD
> + cd /root/rpmbuild/BUILD
> + rm -rf linux
> + /usr/bin/gzip -dc /root/rpmbuild/SOURCES/linux.tar.gz
> + /usr/bin/tar -xof -
> + STATUS=0
> + '[' 0 -ne 0 ']'
> + cd linux
> /var/tmp/rpm-tmp.yDFEga: line 37: cd: linux: No such file or directory
> error: Bad exit status from /var/tmp/rpm-tmp.yDFEga (%prep)
>
>
> RPM build errors:
> Bad exit status from /var/tmp/rpm-tmp.yDFEga (%prep)
> make[1]: *** [scripts/Makefile.package:69: rpm-pkg] Error 1
> make: *** [Makefile:1656: rpm-pkg] Error 2
Thanks for the report. It should/will be fixed with
https://lore.kernel.org/linux-kbuild/20230419170424.78688-1-masahiroy@kernel.org/
Kind regards,
Nicolas
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-20 9:06 ` Nicolas Schier
@ 2023-04-20 9:19 ` Leon Romanovsky
2023-04-20 12:31 ` Masahiro Yamada
0 siblings, 1 reply; 12+ messages in thread
From: Leon Romanovsky @ 2023-04-20 9:19 UTC (permalink / raw)
To: Masahiro Yamada, Nathan Chancellor
Cc: Nicolas Schier, Tariq Toukan, linux-kbuild, linux-kernel,
Nick Desaulniers, Nick Terrell, dalevi, Gal Pressman,
Saeed Mahameed, Tariq Toukan, Linus Torvalds
On Thu, Apr 20, 2023 at 11:06:06AM +0200, Nicolas Schier wrote:
> On Thu, Apr 20, 2023 at 11:54:34AM +0300, Tariq Toukan wrote:
> >
> >
> > On 07/04/2023 21:12, Nathan Chancellor wrote:
> > > On Fri, Apr 07, 2023 at 07:16:29PM +0900, Masahiro Yamada wrote:
> > > > Commit 05e96e96a315 ("kbuild: use git-archive for source package
> > > > creation") split the compression as a separate step to factor out
> > > > the common build rules.
> > > >
> > > > With the previous commit, we got back to the situation where
> > > > compressed source tarballs are created by a single rule.
> > > > There is no reason to keep the separate compression rules.
> > > >
> > > > Generate the comressed tar packages directly.
> > > >
> > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > >
> > > Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> > >
> >
> > Hi,
> >
> > We started seeing the failure below in rc7.
> > We narrowed it down to your patches:
> >
> > 3c65a2704cdd kbuild: do not create intermediate *.tar for tar packages
> > f8d94c4e403c kbuild: do not create intermediate *.tar for source tarballs
> > f6d8283549bc kbuild: merge cmd_archive_linux and cmd_archive_perf
> > aa7d233f45b4 kbuild: give up untracked files for source package builds
<...>
> > RPM build errors:
> > Bad exit status from /var/tmp/rpm-tmp.yDFEga (%prep)
> > make[1]: *** [scripts/Makefile.package:69: rpm-pkg] Error 1
> > make: *** [Makefile:1656: rpm-pkg] Error 2
>
> Thanks for the report. It should/will be fixed with
> https://lore.kernel.org/linux-kbuild/20230419170424.78688-1-masahiroy@kernel.org/
Thanks for the prompt response.
I have a general question, why commits listed by Tariq were not delayed to merge window?
Only one of them has Fixes line, but even that patch doesn't talk about
error, but code refactoring "To simplify the code, ...".
Thanks
>
> Kind regards,
> Nicolas
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-20 9:19 ` Leon Romanovsky
@ 2023-04-20 12:31 ` Masahiro Yamada
0 siblings, 0 replies; 12+ messages in thread
From: Masahiro Yamada @ 2023-04-20 12:31 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Nathan Chancellor, Nicolas Schier, Tariq Toukan, linux-kbuild,
linux-kernel, Nick Desaulniers, Nick Terrell, dalevi,
Gal Pressman, Saeed Mahameed, Tariq Toukan, Linus Torvalds
On Thu, Apr 20, 2023 at 6:19 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Thu, Apr 20, 2023 at 11:06:06AM +0200, Nicolas Schier wrote:
> > On Thu, Apr 20, 2023 at 11:54:34AM +0300, Tariq Toukan wrote:
> > >
> > >
> > > On 07/04/2023 21:12, Nathan Chancellor wrote:
> > > > On Fri, Apr 07, 2023 at 07:16:29PM +0900, Masahiro Yamada wrote:
> > > > > Commit 05e96e96a315 ("kbuild: use git-archive for source package
> > > > > creation") split the compression as a separate step to factor out
> > > > > the common build rules.
> > > > >
> > > > > With the previous commit, we got back to the situation where
> > > > > compressed source tarballs are created by a single rule.
> > > > > There is no reason to keep the separate compression rules.
> > > > >
> > > > > Generate the comressed tar packages directly.
> > > > >
> > > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > > >
> > > > Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> > > >
> > >
> > > Hi,
> > >
> > > We started seeing the failure below in rc7.
> > > We narrowed it down to your patches:
> > >
> > > 3c65a2704cdd kbuild: do not create intermediate *.tar for tar packages
> > > f8d94c4e403c kbuild: do not create intermediate *.tar for source tarballs
> > > f6d8283549bc kbuild: merge cmd_archive_linux and cmd_archive_perf
Strictly speaking, these are not bugs,
but gave a bad user experience.
(build slowness and extra disk space consumed).
I got a complaint about the intermediate *.tar file.
https://lore.kernel.org/linux-kbuild/20230406152540.8207-1-youling257@gmail.com/
I noticed fixing it was not difficult, so I merged.
(but, apparently I introduced another regression, sorry about that)
> > > aa7d233f45b4 kbuild: give up untracked files for source package builds
>
> <...>
>
> > > RPM build errors:
> > > Bad exit status from /var/tmp/rpm-tmp.yDFEga (%prep)
> > > make[1]: *** [scripts/Makefile.package:69: rpm-pkg] Error 1
> > > make: *** [Makefile:1656: rpm-pkg] Error 2
> >
> > Thanks for the report. It should/will be fixed with
> > https://lore.kernel.org/linux-kbuild/20230419170424.78688-1-masahiroy@kernel.org/
>
> Thanks for the prompt response.
>
> I have a general question, why commits listed by Tariq were not delayed to merge window?
>
> Only one of them has Fixes line, but even that patch doesn't talk about
> error, but code refactoring "To simplify the code, ...".
>
> Thanks
>
> >
> > Kind regards,
> > Nicolas
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages
2023-04-07 10:16 ` [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages Masahiro Yamada
2023-04-07 18:12 ` Nathan Chancellor
@ 2023-04-11 0:33 ` Masahiro Yamada
1 sibling, 0 replies; 12+ messages in thread
From: Masahiro Yamada @ 2023-04-11 0:33 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-kernel, Nathan Chancellor, Nick Desaulniers, Nick Terrell,
Nicolas Schier
On Fri, Apr 7, 2023 at 7:16 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Commit 05e96e96a315 ("kbuild: use git-archive for source package
> creation") split the compression as a separate step to factor out
> the common build rules.
>
> With the previous commit, we got back to the situation where
> compressed source tarballs are created by a single rule.
> There is no reason to keep the separate compression rules.
>
> Generate the comressed tar packages directly.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
> scripts/Makefile.package | 27 +++++++++------------------
> 1 file changed, 9 insertions(+), 18 deletions(-)
>
> diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> index 7707975f729b..e0e18d7dfbd5 100644
> --- a/scripts/Makefile.package
> +++ b/scripts/Makefile.package
> @@ -2,7 +2,6 @@
> # Makefile for the different targets used to generate full packages of a kernel
>
> include $(srctree)/scripts/Kbuild.include
> -include $(srctree)/scripts/Makefile.lib
I noticed a bug.
I will keep this include directive.
Otherwise, perf-tar*-src-pkg targets would be broken.
'cmd_copy' is still used.
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] kbuild: merge cmd_archive_linux and cmd_archive_perf
2023-04-07 10:16 [PATCH 1/3] kbuild: merge cmd_archive_linux and cmd_archive_perf Masahiro Yamada
2023-04-07 10:16 ` [PATCH 2/3] kbuild: do not create intermediate *.tar for source tarballs Masahiro Yamada
2023-04-07 10:16 ` [PATCH 3/3] kbuild: do not create intermediate *.tar for tar packages Masahiro Yamada
@ 2023-04-07 18:00 ` Nathan Chancellor
2 siblings, 0 replies; 12+ messages in thread
From: Nathan Chancellor @ 2023-04-07 18:00 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Nick Desaulniers, Nicolas Schier
On Fri, Apr 07, 2023 at 07:16:27PM +0900, Masahiro Yamada wrote:
> The two commands, cmd_archive_linux and cmd_archive_perf, are similar.
> Merge them to make it easier to add more changes to the git-archive
> command.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> ---
>
> scripts/Makefile.package | 19 +++++++++----------
> 1 file changed, 9 insertions(+), 10 deletions(-)
>
> diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> index 61f72eb8d9be..a205617730c6 100644
> --- a/scripts/Makefile.package
> +++ b/scripts/Makefile.package
> @@ -57,16 +57,17 @@ check-git:
> false; \
> fi
>
> +quiet_cmd_archive = ARCHIVE $@
> + cmd_archive = git -C $(srctree) archive \
> + --output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
> +
> # Linux source tarball
> # ---------------------------------------------------------------------------
>
> -quiet_cmd_archive_linux = ARCHIVE $@
> - cmd_archive_linux = \
> - git -C $(srctree) archive --output=$$(realpath $@) --prefix=$(basename $@)/ $$(cat $<)
> -
> targets += linux.tar
> +linux.tar: archive-args = $$(cat $<)
> linux.tar: .tmp_HEAD FORCE
> - $(call if_changed,archive_linux)
> + $(call if_changed,archive)
>
> # rpm-pkg
> # ---------------------------------------------------------------------------
> @@ -180,16 +181,14 @@ quiet_cmd_perf_version_file = GEN $@
> .tmp_perf/PERF-VERSION-FILE: .tmp_HEAD $(srctree)/tools/perf/util/PERF-VERSION-GEN | .tmp_perf
> $(call cmd,perf_version_file)
>
> -quiet_cmd_archive_perf = ARCHIVE $@
> - cmd_archive_perf = \
> - git -C $(srctree) archive --output=$$(realpath $@) --prefix=$(basename $@)/ \
> - --add-file=$$(realpath $(word 2, $^)) \
> +perf-archive-args = --add-file=$$(realpath $(word 2, $^)) \
> --add-file=$$(realpath $(word 3, $^)) \
> $$(cat $(word 2, $^))^{tree} $$(cat $<)
>
> targets += perf-$(KERNELVERSION).tar
> +perf-$(KERNELVERSION).tar: archive-args = $(perf-archive-args)
> perf-$(KERNELVERSION).tar: tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
> - $(call if_changed,archive_perf)
> + $(call if_changed,archive)
>
> PHONY += perf-tar-src-pkg
> perf-tar-src-pkg: perf-$(KERNELVERSION).tar
> --
> 2.37.2
>
^ permalink raw reply [flat|nested] 12+ messages in thread