Re: [PATCH 1/4] xfsprogs: Refactor release scripts to conform to using git archive

Re: [PATCH 1/4] xfsprogs: Refactor release scripts to conform to using git archive

[Ben Myers]

Andrew,

On Mon, Jan 14, 2013 at 12:16:02PM -0600, Andrew Dahl wrote:
> Refactored release scripts to conform to using git archive
> 
> When generating a release, there is a risk of missing necessary
> source files. This is fixed by using git archive, which also
> fixes the lack of conformity between the xfs utilities. As well,
> some files may be stale during packaging. This is fixed with a
> clean at the beginning of release generation.
> 
> Signed-off-by: Andrew Dahl <adahl@sgi.com>

Compared the old 'make dist' with the new one:
# diff old.list new.list
129a130
> xfsprogs-3.1.10/.gitignore

The only difference in the file list is the addition of .gitignore.

> ---
>  Makefile   |   11 ++++++++---
>  release.sh |    3 +++
>  2 files changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 0bdc5e8..e73af88 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -29,6 +29,8 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.gz
>  
>  CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
>  LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
> +SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
> +           m4/ltversion.m4 po/xfsprogs.pot $(CONFIGURE)
>  
>  LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
>  	conftest* built .census install.* install-dev.* *.gz \
> @@ -144,7 +146,10 @@ $(SRCDIR) : $(_FORCE)
>  	rm -fr $@
>  	mkdir -p $@
>  
> -$(SRCTAR) : default $(SRCDIR)
> -	$(Q)$(MAKE) $(MAKEOPTS) source-link
> -	unset TAPE; $(TAR) -cf - $(SRCDIR) | $(ZIP) --best > $@ && \
> +$(SRCTAR) : default
> +	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) \
> +	  -o $(SRCDIR).tar

Older versions of git do not support the -o option.  I changed this to >
$(SRCDIR).tar and it works fine.

> +	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCARC).tar \
						       SRCDIR

With those changes this looks and works fine.

Reviewed-by: Ben Myers <bpm@sgi.com>

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 2/4] xfsdump: Refactor release scripts to conform to using git archive

[Ben Myers]

Andrew,

On Mon, Jan 14, 2013 at 12:16:39PM -0600, Andrew Dahl wrote:
> Refactored release scripts to conform to using git archive
> 
> When generating a release, there is a risk of some files being
> stale, such as configure and the m4/autotools temp files. This
> is fixed with a clean at the beginning of the release generation.
> In addition, there is no uniformity in the current method of
> source tar generation between xfs utilities. Using git archive
> solves this issue accross all utilities.
> 
> Signed-off-by: Andrew Dahl <adahl@sgi.com>

Here is the difference between the old 'make dist' and the new one:

# diff old.list new.list
99a100
> xfsdump-3.1.2/.gitignore
190d190
< xfsdump-3.1.2/po/xfsdump.pot

Adding .gitignore is fine, and xfsdump.pot needn't be in the archive, it is an
output file from gettext.

> diff --git a/Makefile b/Makefile
> index 39881af..23b6a9e 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -29,6 +29,8 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.gz
>  
>  CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
>  LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
> +SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
> +            m4/ltversion.m4 $(CONFIGURE)
>  
>  LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
>  	conftest* built .census install.* install-dev.* *.gz \
> @@ -124,7 +126,10 @@ $(SRCDIR) : $(_FORCE)
>  	rm -fr $@
>  	mkdir -p $@
>  
> -$(SRCTAR) : default $(SRCDIR)
> -	$(Q)$(MAKE) $(MAKEOPTS) source-link
> -	unset TAPE; $(TAR) -cf - $(SRCDIR) | $(ZIP) --best > $@ && \
> +$(SRCTAR) : default
> +	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) \
> +           -o $(SRCDIR).tar

Changed this to 

 > $(SRCDIR).tar

Reviewed-by: Ben Myers <bpm@sgi.com>

Regards,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 3/4] xfstests: Refactor release scripts to conform to using git archive

[Ben Myers]

Andrew,

On Mon, Jan 14, 2013 at 12:17:09PM -0600, Andrew Dahl wrote:

> Refactored release scripts to conform to using git archive
> 
> When generating a release, there is a risk of some files being
> stale, such as configure and the m4/autotools temp files. This
> is fixed with a clean at the beginning of release generation.
> In addition, there is no uniformity in the current method of
> source tar generation between xfs utilities. Using git archive
> solves this issue across all utilities.
> 
> Signed-off-by: Andrew Dahl <adahl@sgi.com>
> ---
>  Makefile   |   20 ++++++++++++++++++++
>  release.sh |   11 +++++++++--
>  2 files changed, 29 insertions(+), 2 deletions(-)
>  mode change 100644 => 100755 release.sh
> 
> diff --git a/Makefile b/Makefile
> index b68ec11..a0aef85 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -39,12 +39,17 @@ ifeq ($(HAVE_BUILDDEFS), yes)
>  include $(TOPDIR)/include/builddefs
>  endif
>  
> +SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.gz
> +
>  TESTS = $(shell sed -n -e '/^[0-9][0-9][0-9]*/s/ .*//p' group)
>  CONFIGURE = configure include/builddefs include/config.h
>  LSRCFILES = configure configure.ac aclocal.m4 README VERSION
>  LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
>  	conftest* check.log check.time
>  
> +ifeq ($(HAVE_BUILDDEFS), yes)
> +LDIRT += $(SRCTAR)
> +endif
>  
>  LIB_SUBDIRS = include lib
>  TOOL_SUBDIRS = ltp src m4
> @@ -102,3 +107,18 @@ install-dev install-lib:
>  realclean distclean: clean
>  	$(Q)rm -f $(LDIRT) $(CONFIGURE)
>  	$(Q)rm -rf autom4te.cache Logs
> +
> +dist: include/builddefs include/config.h default
> +ifeq ($(HAVE_BUILDDEFS), no)
> +	$(Q)$(MAKE) $(MAKEOPTS) -C . $@
> +else
> +	$(Q)$(MAKE) $(MAKEOPTS) $(SRCTAR)
> +endif
> +
> +$(SRCTAR) : default
> +	$(Q)git archive --prefix=$(PKG_NAME)-$(PKG_VERSION)/ --format=tar \
> +	  v$(PKG_VERSION) -o $(PKG_NAME)-$(PKG_VERSION).tar

Again, changed this to
 > $(PKG_NAME)-$(PKG_VERSION).tar

And it worked fine.

Reviewed-by: Ben Myers <bpm@sgi.com>

Regards,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 4/4] dmapi: Refactor release scripts to conform to using git archive

[Ben Myers]

Andrew,

On Mon, Jan 14, 2013 at 12:17:36PM -0600, Andrew Dahl wrote:

> Refactored release scripts to conform to using git archive
> 
> When generating a release, there is a risk of some files being
> stale, such as configure and the m4/autotools temp files.  This
> is fixed with a clean at the beginning of release generation.
> In addition, there is no uniformity in the current method of
> source tar generation between xfs utilities. Using git archive
> solves this issue across all utilities.
> 
> Signed-off-by: Andrew Dahl <adahl@sgi.com>
> ---
>  Makefile       |   19 +++++++++++++++++++
>  Makepkgs       |    4 ++--
>  build/Makefile |   22 +---------------------
>  release.sh     |    5 ++++-
>  4 files changed, 26 insertions(+), 24 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 0bddb07..58e5caf 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -9,6 +9,8 @@ ifeq ($(HAVE_BUILDDEFS), yes)
>  include $(TOPDIR)/include/builddefs
>  endif
>  
> +SRCTAR=$(PKG_NAME)-$(PKG_VERSION).tar.gz
> +
>  CONFIGURE = aclocal.m4 configure config.guess config.sub configure install-sh \
>  	    ltmain.sh m4/libtool.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
>  	    m4/ltversion.m4 m4/lt~obsolete.m4
> @@ -84,3 +86,19 @@ distclean: clean
>  
>  realclean: distclean
>  	rm -f $(CONFIGURE)
> +
> +dist: default
> +ifeq ($(HAVE_BUILDDEFS), no)
> +	$(MAKE) $(MAKEOPTS) -C . $@
> +else
> +	$(MAKE) $(MAKEOPTS) $(SRCTAR)
> +	$(MAKE) $(MAKEOPTS) -C build dist
> +endif
> +
> +$(SRCTAR) : default
> +	git archive --prefix=$(PKG_NAME)-$(PKG_VERSION)/ --format=tar \
> +	  v$(PKG_VERSION) -o $(PKG_NAME)-$(PKG_VERSION).tar
		   	  > 

Other than that it looks fine.

# diff old.list new.list 
0a1
> dmapi-2.2.12/
1a3
> dmapi-2.2.12/build/
2a5
> dmapi-2.2.12/build/rpm/
6a10
> dmapi-2.2.12/build/tar/
12a17
> dmapi-2.2.12/debian/
17a23
> dmapi-2.2.12/doc/
22a29,30
> dmapi-2.2.12/.gitignore
> dmapi-2.2.12/include/
30a39
> dmapi-2.2.12/libdm/
49a59
> dmapi-2.2.12/m4/
62a73
> dmapi-2.2.12/man/
63a75
> dmapi-2.2.12/man/man3/

The old source tarball didn't have some directories in it.  That's ok.

Reviewed-by: Ben Myers <bpm@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Ben Myers]

On Mon, Jan 14, 2013 at 12:15:10PM -0600, Andrew Dahl wrote:
> The following four patches fix an issue seen during the last user space
> release where not all source files were included in xfsprogs.  They each
> change xfsprogs, xfstests, xfsdump, and dmapi to all use git archive for
> source tarball generation and add 'make realclean' in the release
> scripts to ensure the configure and m4/autotools temp files are refreshed.

Applied this series.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Dave Chinner]

On Fri, Feb 01, 2013 at 01:09:30PM -0600, Ben Myers wrote:
> On Mon, Jan 14, 2013 at 12:15:10PM -0600, Andrew Dahl wrote:
> > The following four patches fix an issue seen during the last user space
> > release where not all source files were included in xfsprogs.  They each
> > change xfsprogs, xfstests, xfsdump, and dmapi to all use git archive for
> > source tarball generation and add 'make realclean' in the release
> > scripts to ensure the configure and m4/autotools temp files are refreshed.
> 
> Applied this series.

It breaks the 'make deb' command for all the trees.

The debian package build creates a link tree so that it builds
outside the main source tree (e.g. in ./xfsprogs-3.1.10) and as part
of the build it runs 'make dist' to build a tarball from that link
tree. That now fails with:

fatal: current working directory is untracked

coming from the git archive command that is now used to build the
tarball.

Nathan, can the debian package build use git-archive? I'm under the
impression that you have to be able to build debian packages from a
source tarball or a source package, and so the build can't rely on
being inside a VCS controlled tree. That rules out using git archive
for tarball creation...

Also, given that the previous tarball creation used the same link
tree build as the debian package build (i.e. make source-link), that
implies the debian pacakges are going to be missing files using this
technique. Doesn't that mean we have to fix the file list being fed
into the source-link target regardless of whether we use git archive
to build the tarball?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Nathan Scott]

Dave Chinner <david <at> fromorbit.com> writes:

> 
> It breaks the 'make deb' command for all the trees.
> 

(thanks for checking!)

> The debian package build creates a link tree so that it builds
> outside the main source tree (e.g. in ./xfsprogs-3.1.10) and as part
> of the build it runs 'make dist' to build a tarball from that link
> tree. That now fails with:
> 
> fatal: current working directory is untracked
> 
> coming from the git archive command that is now used to build the
> tarball.
> 
> Nathan, can the debian package build use git-archive? I'm under the
> impression that you have to be able to build debian packages from a
> source tarball or a source package, and so the build can't rely on
> being inside a VCS controlled tree. That rules out using git archive
> for tarball creation...
> 

IIRC, the requirement is to be building in a tree that had a basename
of for <package>-<version> - hence the link'd tree build previously.
If that can be maintained somehow, the build should be fine.  Symlinks
did not work very well (IIRC) - the build tools might've used basename
or realpath or something like that, but I do remember trying that.

> Also, given that the previous tarball creation used the same link
> tree build as the debian package build (i.e. make source-link), that
> implies the debian pacakges are going to be missing files using this
> technique. Doesn't that mean we have to fix the file list being fed
> into the source-link target regardless of whether we use git archive
> to build the tarball?

The source tarball definitely needs to have the package-version prefix
for all the files it contains - has a mechanism to acheive that been
found?

cheers.

--
Nathan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git?archive

[Ben Myers]

On Wed, Feb 06, 2013 at 06:28:10AM +0000, Nathan Scott wrote:
> Dave Chinner <david <at> fromorbit.com> writes:
> 
> > 
> > It breaks the 'make deb' command for all the trees.
> > 
> 
> (thanks for checking!)
> 
> > The debian package build creates a link tree so that it builds
> > outside the main source tree (e.g. in ./xfsprogs-3.1.10) and as part
> > of the build it runs 'make dist' to build a tarball from that link
> > tree. That now fails with:
> > 
> > fatal: current working directory is untracked
> > 
> > coming from the git archive command that is now used to build the
> > tarball.
> > 
> > Nathan, can the debian package build use git-archive? I'm under the
> > impression that you have to be able to build debian packages from a
> > source tarball or a source package, and so the build can't rely on
> > being inside a VCS controlled tree. That rules out using git archive
> > for tarball creation...
> > 
> 
> IIRC, the requirement is to be building in a tree that had a basename
> of for <package>-<version> - hence the link'd tree build previously.
> If that can be maintained somehow, the build should be fine.  Symlinks
> did not work very well (IIRC) - the build tools might've used basename
> or realpath or something like that, but I do remember trying that.
> 
> > Also, given that the previous tarball creation used the same link
> > tree build as the debian package build (i.e. make source-link), that
> > implies the debian pacakges are going to be missing files using this
> > technique. Doesn't that mean we have to fix the file list being fed
> > into the source-link target regardless of whether we use git archive
> > to build the tarball?
> 
> The source tarball definitely needs to have the package-version prefix
> for all the files it contains - has a mechanism to acheive that been
> found?

It has... with a combo of git archive --prefix and tar --transform.  I'll see
what I can do to get 'make deb' going again.

Thanks,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git?archive

[Dave Chinner]

On Wed, Feb 06, 2013 at 06:28:10AM +0000, Nathan Scott wrote:
> Dave Chinner <david <at> fromorbit.com> writes:
> 
> > 
> > It breaks the 'make deb' command for all the trees.
> > 
> 
> (thanks for checking!)

I didn't - my CI system caught it immediately once I switched
everything back on after being away for a week at LCA. It builds
debs and ships them to VMs, installs them and then runs various
tests after every upstream repository check in that occurs.

It was when I checked the RSS feeds coming back from the CI system
that I noticed that it was busted and started looking into the
problem....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Ben Myers]

On Wed, Feb 06, 2013 at 12:15:47PM +1100, Dave Chinner wrote:
> On Fri, Feb 01, 2013 at 01:09:30PM -0600, Ben Myers wrote:
> > On Mon, Jan 14, 2013 at 12:15:10PM -0600, Andrew Dahl wrote:
> > > The following four patches fix an issue seen during the last user space
> > > release where not all source files were included in xfsprogs.  They each
> > > change xfsprogs, xfstests, xfsdump, and dmapi to all use git archive for
> > > source tarball generation and add 'make realclean' in the release
> > > scripts to ensure the configure and m4/autotools temp files are refreshed.
> > 
> > Applied this series.
> 
> It breaks the 'make deb' command for all the trees.

xfsprogs: update 'make deb' to use tarball

This patch changes the build process so that 'make deb' uses the same
process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .git_census
  This is needed so that you can create a tarball in a bare release
  tree, when .git is not available.

* Modify the SRCTAR target to include files from .git_census and use tar
  instead of git archive.

* Modify the SRCTARINC files to include .git_census, and include 
  .git_census in the 'make realclean' target.

* 'make deb' now creates unsigned packages by default, 'make debsign'
  creates signed packages.

* remove the 'make source-link' target.

Signed-off-by: Ben Myers <bpm@sgi.com>

---
 Makefile           |   29 ++++++++++++++++-------------
 include/buildrules |   11 -----------
 2 files changed, 16 insertions(+), 24 deletions(-)

Index: b/Makefile
===================================================================
--- a/Makefile	2013-02-13 16:55:18.451946936 -0600
+++ b/Makefile	2013-02-13 18:39:16.121985626 -0600
@@ -30,8 +30,7 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
 SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
-           m4/ltversion.m4 po/xfsprogs.pot $(CONFIGURE)
-
+           m4/ltversion.m4 po/xfsprogs.pot .git_census $(CONFIGURE)
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \
 	autom4te.cache/* libtool include/builddefs include/platform_defs.h
@@ -120,7 +119,7 @@ distclean: clean
 	$(Q)rm -f $(LDIRT)
 
 realclean: distclean
-	$(Q)rm -f $(CONFIGURE)
+	$(Q)rm -f $(CONFIGURE) .git_census
 
 #
 # All this gunk is to allow for a make dist on an unconfigured tree
@@ -137,18 +136,22 @@ ifeq ($(HAVE_BUILDDEFS), no)
 	$(Q)$(MAKE) $(MAKEOPTS) -C . $@
 else
 	$(Q)$(MAKE) $(MAKEOPTS) $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) -C po
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
-	$(Q)cd $(SRCDIR) && dpkg-buildpackage
+	$(Q)cd $(SRCDIR) && dpkg-buildpackage -us -uc
 endif
 
-$(SRCDIR) : $(_FORCE)
+debsign: deb
+	debsign
+
+$(SRCDIR) : $(_FORCE) $(SRCTAR)
 	rm -fr $@
-	mkdir -p $@
+	$(Q)$(TAR) -zxvf $(SRCTAR)
 
-$(SRCTAR) : default
-	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) > $(SRCDIR).tar 
-	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCDIR).tar \
-	   $(SRCTARINC) 
-	$(Q)$(ZIP) $(SRCDIR).tar
+$(SRCTAR) : default $(SRCTARINC) .git_census
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -zcf $(SRCDIR).tar.gz  \
+	   `cat .git_census` $(SRCTARINC)
 	echo Wrote: $@
+
+.git_census: $(_FORCE)
+	$(Q)if test -d .git; then \
+	  git ls-files > .git_census && echo "new .git_census"; \
+	fi
Index: b/include/buildrules
===================================================================
--- a/include/buildrules	2013-02-13 17:30:04.181945855 -0600
+++ b/include/buildrules	2013-02-13 17:30:24.161985853 -0600
@@ -23,17 +23,6 @@ $(SUBDIRS):
 	$(Q)$(MAKE) $(MAKEOPTS) -q -C $@ || $(MAKE) $(MAKEOPTS) -C $@
 endif
 
-source-link: 
-	@test -z "$$DIR" && DIR="."; \
-	for f in `echo $(SRCFILES) $(SUBDIRS) $(POTHEAD)`; do \
-	    if test -d $$f ; then \
-		mkdir $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-		$(MAKEF) DIR=$$DIR/$$f -C $$f $@ || exit $$?; \
-	    else \
-		ln $$f $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-	    fi; \
-	done
-
 #
 # Standard targets
 #

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Nathan Scott]

Hi Ben,

----- Original Message -----
> On Wed, Feb 06, 2013 at 12:15:47PM +1100, Dave Chinner wrote:
> > ...
> > It breaks the 'make deb' command for all the trees.
> 
> xfsprogs: update 'make deb' to use tarball
> 
> This patch changes the build process so that 'make deb' uses the same
> process of creating a source tree as the release script.
> 
> * Add a list of files which go in the release tarball in .git_census
>   This is needed so that you can create a tarball in a bare release
>   tree, when .git is not available.
> 

I think you'll need .git_census in .gitignore (ala configure).
Possibly remove the underscore for naming consistency - *shrug*.

> 
> * 'make deb' now creates unsigned packages by default, 'make debsign'
>   creates signed packages.
> 

Ehrm - why?  Everything else in your patch worked, but this part broke
(which suggests a larger problem, in that this build path is no longer
checked on every build) ...

debsign: Can't find or can't read changes file !

> +debsign: deb
> +	debsign
> +

(That's not valid usage, FWIW).

I would recommend just removing that change in behaviour, "make deb"
was fine as it was (for me anyway, and evidently for Dave too).  Best
to go secure-by-default and not change this.

thanks!

--
Nathan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Ben Myers]

Hey Nathan,

On Wed, Feb 13, 2013 at 09:14:37PM -0500, Nathan Scott wrote:
> ----- Original Message -----
> > On Wed, Feb 06, 2013 at 12:15:47PM +1100, Dave Chinner wrote:
> > > ...
> > > It breaks the 'make deb' command for all the trees.
> > 
> > xfsprogs: update 'make deb' to use tarball
> > 
> > This patch changes the build process so that 'make deb' uses the same
> > process of creating a source tree as the release script.
> > 
> > * Add a list of files which go in the release tarball in .git_census
> >   This is needed so that you can create a tarball in a bare release
> >   tree, when .git is not available.
> > 
> 
> I think you'll need .git_census in .gitignore (ala configure).
> Possibly remove the underscore for naming consistency - *shrug*.

Will do.  Thanks.

> 
> > 
> > * 'make deb' now creates unsigned packages by default, 'make debsign'
> >   creates signed packages.
> > 
> 
> Ehrm - why?  Everything else in your patch worked, but this part broke
> (which suggests a larger problem, in that this build path is no longer
> checked on every build) ...
> 
> debsign: Can't find or can't read changes file !
> 
> > +debsign: deb
> > +	debsign
> > +
> 
> (That's not valid usage, FWIW).
>
> I would recommend just removing that change in behaviour, "make deb"
> was fine as it was (for me anyway, and evidently for Dave too).  Best
> to go secure-by-default and not change this.

'make deb' failed for me like this:

signfile xfsprogs_3.1.9.dsc
gpg: skipped "Nathan Scott <nathans@debian.org>": secret key not available
gpg: [stdin]: clearsign failed: secret key not available

dpkg-genchanges  >../xfsprogs_3.1.9_amd64.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload; Debian-native package (full source is included)
dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [deb] Error 1

Secure-by-default is a great idea but it seems that in this case it also
requires that anyone who wants to do a build for debian has to do it on a
machine with a keyring containing your secret key. ;)

I think it is also reasonable for devs to be able to build test debs on
machines that have empty keyrings.  An unsigned 'make deb' does return 0.  I'd
like this to suit everyone's needs if it can.  I'll split this part out into a
separate patch and try to figure something out later.

Thanks,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

xfsprogs: update 'make deb' to use tarball

[Ben Myers]

This patch changes the build process so that 'make deb' uses the same
process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus
  This is needed so that you can create a tarball in a bare release
  tree, when .git is not available.

* Modify the SRCTAR target to include files from .git_census and use tar
  instead of git archive.

* Modify the SRCTARINC files to include .git_census, and include 
  .git_census in the 'make realclean' target.

* remove the 'make source-link' target.

Signed-off-by: Ben Myers <bpm@sgi.com>

---

v2: don't muck with signing the deb in this patch, rename .git_census to
.gitcensus, add .gitcensus to .gitignore.


 .gitignore         |    1 +
 Makefile           |   24 ++++++++++++------------
 include/buildrules |   11 -----------
 3 files changed, 13 insertions(+), 23 deletions(-)

Index: b/.gitignore
===================================================================
--- a/.gitignore	2013-02-14 10:40:25.861946781 -0600
+++ b/.gitignore	2013-02-14 10:40:39.352005790 -0600
@@ -5,6 +5,7 @@
 
 # build system
 .census
+.gitcensus
 /include/platform_defs.h
 /include/builddefs
 /install-sh
Index: b/Makefile
===================================================================
--- a/Makefile	2013-02-14 09:43:02.411946730 -0600
+++ b/Makefile	2013-02-14 10:41:15.151360977 -0600
@@ -30,8 +30,7 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
 SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
-           m4/ltversion.m4 po/xfsprogs.pot $(CONFIGURE)
-
+           m4/ltversion.m4 po/xfsprogs.pot .gitcensus $(CONFIGURE)
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \
 	autom4te.cache/* libtool include/builddefs include/platform_defs.h
@@ -120,7 +119,7 @@ distclean: clean
 	$(Q)rm -f $(LDIRT)
 
 realclean: distclean
-	$(Q)rm -f $(CONFIGURE)
+	$(Q)rm -f $(CONFIGURE) .gitcensus
 
 #
 # All this gunk is to allow for a make dist on an unconfigured tree
@@ -137,18 +136,19 @@ ifeq ($(HAVE_BUILDDEFS), no)
 	$(Q)$(MAKE) $(MAKEOPTS) -C . $@
 else
 	$(Q)$(MAKE) $(MAKEOPTS) $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) -C po
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
 	$(Q)cd $(SRCDIR) && dpkg-buildpackage
 endif
 
-$(SRCDIR) : $(_FORCE)
+$(SRCDIR) : $(_FORCE) $(SRCTAR)
 	rm -fr $@
-	mkdir -p $@
+	$(Q)$(TAR) -zxvf $(SRCTAR)
 
-$(SRCTAR) : default
-	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) > $(SRCDIR).tar 
-	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCDIR).tar \
-	   $(SRCTARINC) 
-	$(Q)$(ZIP) $(SRCDIR).tar
+$(SRCTAR) : default $(SRCTARINC) .gitcensus
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -zcf $(SRCDIR).tar.gz  \
+	   `cat .gitcensus` $(SRCTARINC)
 	echo Wrote: $@
+
+.gitcensus: $(_FORCE)
+	$(Q)if test -d .git; then \
+	  git ls-files > .gitcensus && echo "new .gitcensus"; \
+	fi
Index: b/include/buildrules
===================================================================
--- a/include/buildrules	2013-02-14 09:43:02.421945975 -0600
+++ b/include/buildrules	2013-02-14 09:44:13.291947757 -0600
@@ -23,17 +23,6 @@ $(SUBDIRS):
 	$(Q)$(MAKE) $(MAKEOPTS) -q -C $@ || $(MAKE) $(MAKEOPTS) -C $@
 endif
 
-source-link: 
-	@test -z "$$DIR" && DIR="."; \
-	for f in `echo $(SRCFILES) $(SUBDIRS) $(POTHEAD)`; do \
-	    if test -d $$f ; then \
-		mkdir $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-		$(MAKEF) DIR=$$DIR/$$f -C $$f $@ || exit $$?; \
-	    else \
-		ln $$f $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-	    fi; \
-	done
-
 #
 # Standard targets
 #

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfsprogs: update 'make deb' to use tarball

[Nathan Scott]

----- Original Message -----
> This patch changes the build process so that 'make deb' uses the same
> process of creating a source tree as the release script.
> 
> * Add a list of files which go in the release tarball in .gitcensus
>   This is needed so that you can create a tarball in a bare release
>   tree, when .git is not available.
> 
> * Modify the SRCTAR target to include files from .git_census and use
> tar
>   instead of git archive.
> 
> * Modify the SRCTARINC files to include .git_census, and include
>   .git_census in the 'make realclean' target.
> 
> * remove the 'make source-link' target.
> 
> Signed-off-by: Ben Myers <bpm@sgi.com>
> 

Reviewed-by: Nathan Scott <nathans@debian.org>

thanks!

--
Nathan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfsprogs: update 'make deb' to use tarball

[Dave Chinner]

On Thu, Feb 14, 2013 at 10:54:09AM -0600, Ben Myers wrote:
> This patch changes the build process so that 'make deb' uses the same
> process of creating a source tree as the release script.
> 
> * Add a list of files which go in the release tarball in .gitcensus
>   This is needed so that you can create a tarball in a bare release
>   tree, when .git is not available.
> 
> * Modify the SRCTAR target to include files from .git_census and use tar
>   instead of git archive.
> 
> * Modify the SRCTARINC files to include .git_census, and include 
>   .git_census in the 'make realclean' target.
> 
> * remove the 'make source-link' target.
> 
> Signed-off-by: Ben Myers <bpm@sgi.com>

Works fine here.

Reviewed-by: Dave Chinner <dchinner@redhat.com>

-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

xfsdump: update 'make deb' to use tarball archive

[Ben Myers]

This patch changes the build process so that 'make deb' uses the same
process of creating a source tree as the release script.

* Add a list of files which go in the release tarball in .gitcensus
  This is needed so that you can create a tarball in a bare release
  tree, when .git is not available.

* Modify the SRCTAR target to include files from .gitcensus and use tar
  instead of git archive.

* Modify the SRCTARINC files to include .gitcensus, and include
  .gitcensus in the 'make realclean' target.

* remove the 'make source-link' target.

Signed-off-by: Ben Myers <bpm@sgi.com>

---
 .gitignore         |    1 +
 Makefile           |   23 ++++++++++++-----------
 include/buildrules |   11 -----------
 3 files changed, 13 insertions(+), 22 deletions(-)

Index: b/.gitignore
===================================================================
--- a/.gitignore	2013-02-14 10:55:13.061985803 -0600
+++ b/.gitignore	2013-02-14 10:55:49.061340568 -0600
@@ -4,6 +4,7 @@
 
 # build system
 .census
+.gitcensus
 /include/config.h
 /include/builddefs
 /install-sh
Index: b/Makefile
===================================================================
--- a/Makefile	2013-02-14 10:55:06.961977139 -0600
+++ b/Makefile	2013-02-14 10:59:19.131340450 -0600
@@ -30,7 +30,7 @@ SRCTAR = $(PKG_NAME)-$(PKG_VERSION).tar.
 CONFIGURE = aclocal.m4 configure config.guess config.sub install-sh ltmain.sh
 LSRCFILES = configure.ac release.sh README VERSION $(CONFIGURE)
 SRCTARINC = m4/libtool.m4 m4/lt~obsolete.m4 m4/ltoptions.m4 m4/ltsugar.m4 \
-            m4/ltversion.m4 $(CONFIGURE)
+            m4/ltversion.m4 .gitcensus $(CONFIGURE)
 
 LDIRT = config.log .ltdep .dep config.status config.cache confdefs.h \
 	conftest* built .census install.* install-dev.* *.gz \
@@ -100,7 +100,7 @@ distclean: clean
 	rm -f $(LDIRT)
 
 realclean: distclean
-	rm -f $(CONFIGURE)
+	rm -f $(CONFIGURE) .gitcensus
 
 #
 # All this gunk is to allow for a make dist on an unconfigured tree
@@ -117,18 +117,19 @@ ifeq ($(HAVE_BUILDDEFS), no)
 	$(MAKE) $(MAKEOPTS) -C . $@
 else
 	$(Q)$(MAKE) $(MAKEOPTS) $(SRCDIR)
-	$(Q)$(MAKE) $(MAKEOPTS) -C po
-	$(Q)$(MAKE) $(MAKEOPTS) source-link
 	$(Q)cd $(SRCDIR) && dpkg-buildpackage
 endif
 
-$(SRCDIR) : $(_FORCE)
+$(SRCDIR) : $(_FORCE) $(SRCTAR)
 	rm -fr $@
-	mkdir -p $@
+	$(Q)$(TAR) -zxvf $(SRCTAR)
 
-$(SRCTAR) : default
-	$(Q)git archive --prefix=$(SRCDIR)/ --format=tar v$(PKG_VERSION) > $(SRCDIR).tar
-	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -rf $(SRCDIR).tar \
-	  $(SRCTARINC)
-	$(Q)$(ZIP) $(SRCDIR).tar
+$(SRCTAR) : default $(SRCTARINC) .gitcensus
+	$(Q)$(TAR) --transform "s,^,$(SRCDIR)/," -zcf $(SRCDIR).tar.gz \
+	  `cat .gitcensus` $(SRCTARINC)
 	echo Wrote: $@
+
+.gitcensus: $(_FORCE)
+	$(Q) if test -d .git; then \
+	  git ls-files > .gitcensus && echo "new .gitcensus"; \
+	fi
Index: b/include/buildrules
===================================================================
--- a/include/buildrules	2013-02-14 10:59:26.221945695 -0600
+++ b/include/buildrules	2013-02-14 10:59:39.321985640 -0600
@@ -23,17 +23,6 @@ $(SUBDIRS):
 	$(Q)$(MAKE) $(MAKEOPTS) -q -C $@ || $(MAKE) $(MAKEOPTS) -C $@
 endif
 
-source-link: 
-	@test -z "$$DIR" && DIR="."; \
-	for f in `echo $(SRCFILES) $(SUBDIRS) $(POTHEAD)`; do \
-	    if test -d $$f ; then \
-		mkdir $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-		$(MAKEF) DIR=$$DIR/$$f -C $$f $@ || exit $$?; \
-	    else \
-		ln $$f $(TOPDIR)/$(PKG_NAME)-$(PKG_VERSION)/$$DIR/$$f || exit $$?; \
-	    fi; \
-	done
-
 #
 # Standard targets
 #

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfsdump: update 'make deb' to use tarball archive

[Nathan Scott]

----- Original Message -----
> This patch changes the build process so that 'make deb' uses the same
> process of creating a source tree as the release script.
> 
> * Add a list of files which go in the release tarball in .gitcensus
>   This is needed so that you can create a tarball in a bare release
>   tree, when .git is not available.
> 
> * Modify the SRCTAR target to include files from .gitcensus and use
> tar
>   instead of git archive.
> 
> * Modify the SRCTARINC files to include .gitcensus, and include
>   .gitcensus in the 'make realclean' target.
> 
> * remove the 'make source-link' target.
> 
> Signed-off-by: Ben Myers <bpm@sgi.com>
> 

Reviewed-by: Nathan Scott <nathans@debian.org>

thanks!

--
Nathan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfsdump: update 'make deb' to use tarball archive

[Dave Chinner]

On Thu, Feb 14, 2013 at 11:31:20AM -0600, Ben Myers wrote:
> This patch changes the build process so that 'make deb' uses the same
> process of creating a source tree as the release script.
> 
> * Add a list of files which go in the release tarball in .gitcensus
>   This is needed so that you can create a tarball in a bare release
>   tree, when .git is not available.
> 
> * Modify the SRCTAR target to include files from .gitcensus and use tar
>   instead of git archive.
> 
> * Modify the SRCTARINC files to include .gitcensus, and include
>   .gitcensus in the 'make realclean' target.
> 
> * remove the 'make source-link' target.
> 
> Signed-off-by: Ben Myers <bpm@sgi.com>

Works fine.

Reviewed-by: Dave Chinner <dchinner@redhat.com>

-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Nathan Scott]

----- Original Message -----
> ...
> 'make deb' failed for me like this:
> 
> signfile xfsprogs_3.1.9.dsc
> gpg: skipped "Nathan Scott <nathans@debian.org>": secret key not
> available
> gpg: [stdin]: clearsign failed: secret key not available
> 
> dpkg-genchanges  >../xfsprogs_3.1.9_amd64.changes
> dpkg-genchanges: including full source code in upload
> dpkg-buildpackage: full upload; Debian-native package (full source is
> included)
> dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
> make: *** [deb] Error 1
> 
> Secure-by-default is a great idea but it seems that in this case it
> also
> requires that anyone who wants to do a build for debian has to do it
> on a
> machine with a keyring containing your secret key. ;)
> 

Awkward.  :)

> I think it is also reasonable for devs to be able to build test debs

*nod* ... not sure what trickery Dave had used.  But, perhaps a better
way to tackle this would be similar to the way the build allows extra
configure options to be passed in.

include/builddefs: configure
        ./configure $$LOCAL_CONFIGURE_OPTIONS

so, perhaps something like this...

# LOCAL_BUILDPACKAGE_OPTIONS="-us -uc" for unsigned debs
deb: include/builddefs include/platform_defs.h
...
        $(Q)cd $(SRCDIR) && dpkg-buildpackage $$LOCAL_BUILDPACKAGE_OPTIONS


cheers.

--
Nathan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Ben Myers]

On Thu, Feb 14, 2013 at 05:25:47PM -0500, Nathan Scott wrote:
> ----- Original Message -----
> > ...
> > 'make deb' failed for me like this:
> > 
> > signfile xfsprogs_3.1.9.dsc
> > gpg: skipped "Nathan Scott <nathans@debian.org>": secret key not
> > available
> > gpg: [stdin]: clearsign failed: secret key not available
> > 
> > dpkg-genchanges  >../xfsprogs_3.1.9_amd64.changes
> > dpkg-genchanges: including full source code in upload
> > dpkg-buildpackage: full upload; Debian-native package (full source is
> > included)
> > dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
> > make: *** [deb] Error 1
> > 
> > Secure-by-default is a great idea but it seems that in this case it
> > also
> > requires that anyone who wants to do a build for debian has to do it
> > on a
> > machine with a keyring containing your secret key. ;)
> > 
> 
> Awkward.  :)

Really I don't mind.  What was your passphrase again? 

> > I think it is also reasonable for devs to be able to build test debs
> 
> *nod* ... not sure what trickery Dave had used.

I tried setting DEB_BUILD_OPTIONS="-us -uc" but no luck.

> But, perhaps a better
> way to tackle this would be similar to the way the build allows extra
> configure options to be passed in.

> 
> include/builddefs: configure
>         ./configure $$LOCAL_CONFIGURE_OPTIONS
> 
> so, perhaps something like this...
> 
> # LOCAL_BUILDPACKAGE_OPTIONS="-us -uc" for unsigned debs
> deb: include/builddefs include/platform_defs.h
> ...
>         $(Q)cd $(SRCDIR) && dpkg-buildpackage $$LOCAL_BUILDPACKAGE_OPTIONS

Gotcha.  I'll take a look.  Thanks for the reviews!

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/4] Refactor release scripts to conform to using git archive

[Dave Chinner]

On Thu, Feb 14, 2013 at 10:24:54AM -0600, Ben Myers wrote:
> Hey Nathan,
> 
> On Wed, Feb 13, 2013 at 09:14:37PM -0500, Nathan Scott wrote:
> > ----- Original Message -----
> > > On Wed, Feb 06, 2013 at 12:15:47PM +1100, Dave Chinner wrote:
> > > > ...
> > > > It breaks the 'make deb' command for all the trees.
> > > 
> > > xfsprogs: update 'make deb' to use tarball
> > > 
> > > This patch changes the build process so that 'make deb' uses the same
> > > process of creating a source tree as the release script.
> > > 
> > > * Add a list of files which go in the release tarball in .git_census
> > >   This is needed so that you can create a tarball in a bare release
> > >   tree, when .git is not available.
> > > 
> > 
> > I think you'll need .git_census in .gitignore (ala configure).
> > Possibly remove the underscore for naming consistency - *shrug*.
> 
> Will do.  Thanks.
> 
> > 
> > > 
> > > * 'make deb' now creates unsigned packages by default, 'make debsign'
> > >   creates signed packages.
> > > 
> > 
> > Ehrm - why?  Everything else in your patch worked, but this part broke
> > (which suggests a larger problem, in that this build path is no longer
> > checked on every build) ...
> > 
> > debsign: Can't find or can't read changes file !
> > 
> > > +debsign: deb
> > > +	debsign
> > > +
> > 
> > (That's not valid usage, FWIW).
> >
> > I would recommend just removing that change in behaviour, "make deb"
> > was fine as it was (for me anyway, and evidently for Dave too).  Best
> > to go secure-by-default and not change this.
> 
> 'make deb' failed for me like this:
> 
> signfile xfsprogs_3.1.9.dsc
> gpg: skipped "Nathan Scott <nathans@debian.org>": secret key not available
> gpg: [stdin]: clearsign failed: secret key not available
> 
> dpkg-genchanges  >../xfsprogs_3.1.9_amd64.changes
> dpkg-genchanges: including full source code in upload
> dpkg-buildpackage: full upload; Debian-native package (full source is included)
> dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
> make: *** [deb] Error 1

Don't know what you are doing differently to me, but make deb is not
even trying to build signed packages on my build machine. It never
has, and I don't ever recall seeing make deb fail because it was
trying to build signed packages in the last 5 years on any of my
machines...

I'm not passing -uc -us to dpkg-buildpackage, so somewhere aling the
line it is working out that it shouldn't be signing the packages
itself...

> I think it is also reasonable for devs to be able to build test debs on
> machines that have empty keyrings.

That's exactly what I have been doing for a long time.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs