packaging issues: some serious, some not so serious

packaging issues: some serious, some not so serious

[Robert P. J. Day]

  there are a few bugs in the packaging targets, both with local
packaging and remote packaging if you've used "O=" for your kernel
build.  first, let's consider local builds.

  consider the packaging makefile scripts/package/Makefile.  these
early lines look entirely superfluous (nothing uses the variable
TAR_IGNORE):

...
# Ignore the following files/directories during tar operation
TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exclude CVS
...

  next, related to my earlier email, if your kernel source happens to
be in its own filesystem so that there's a "lost+found" directory, any
"tar" operation is going to fail due to lack of permissions for that
directory:

$ make rpm-pkg
...
find: ./lost+found: Permission denied
...
tar: kernel-2.6.25rc200015g1309d4e/./lost+found: Cannot open:
Permission denied
tar: Error exit delayed from previous errors
make[1]: *** [rpm-pkg] Error 2
make: *** [rpm-pkg] Error 2
$

  it seems to be sufficient to apply the following patch:

diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 5e32607..d3307f1 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -39,7 +39,7 @@ $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile
 rpm-pkg rpm: $(objtree)/kernel.spec FORCE
        $(MAKE) clean
        $(PREV) ln -sf $(srctree) $(KERNELPATH)
-       $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
+       $(PREV) tar -cz --exclude lost+found $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
        $(PREV) rm $(KERNELPATH)

        set -e; \


but it's probable that a number of other operations need to take a
"lost+found" directory into account; that is, anything involving
"tar".

  observations on remote packaging to come.

rday
--


========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: packaging issues: some serious, some not so serious

[Sam Ravnborg]

On Mon, Feb 18, 2008 at 11:35:30AM -0500, Robert P. J. Day wrote:
> 
>   there are a few bugs in the packaging targets, both with local
> packaging and remote packaging if you've used "O=" for your kernel
> build.  first, let's consider local builds.
> 
>   consider the packaging makefile scripts/package/Makefile.  these
> early lines look entirely superfluous (nothing uses the variable
> TAR_IGNORE):
> 
> ...
> # Ignore the following files/directories during tar operation
> TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --exclude CVS
> ...
> 
>   next, related to my earlier email, if your kernel source happens to
> be in its own filesystem so that there's a "lost+found" directory, any
> "tar" operation is going to fail due to lack of permissions for that
> directory:
> 
> $ make rpm-pkg
> ...
> find: ./lost+found: Permission denied
> ...
> tar: kernel-2.6.25rc200015g1309d4e/./lost+found: Cannot open:
> Permission denied
> tar: Error exit delayed from previous errors
> make[1]: *** [rpm-pkg] Error 2
> make: *** [rpm-pkg] Error 2
> $
> 
>   it seems to be sufficient to apply the following patch:
> 
> diff --git a/scripts/package/Makefile b/scripts/package/Makefile
> index 5e32607..d3307f1 100644
> --- a/scripts/package/Makefile
> +++ b/scripts/package/Makefile
> @@ -39,7 +39,7 @@ $(objtree)/kernel.spec: $(MKSPEC) $(srctree)/Makefile
>  rpm-pkg rpm: $(objtree)/kernel.spec FORCE
>         $(MAKE) clean
>         $(PREV) ln -sf $(srctree) $(KERNELPATH)
> -       $(PREV) tar -cz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
> +       $(PREV) tar -cz --exclude lost+found $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/.
>         $(PREV) rm $(KERNELPATH)
> 
I would much rather you persuaded your earlier idea list all
the directories to visit. This list is known and easy to maintain
whereas a blacklist like we have now only will grow bigger over time.

We could add something like:
diff --git a/Makefile b/Makefile
index 0d585c0..fb5bbc3 100644
--- a/Makefile
+++ b/Makefile
@@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
 RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
 export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex

+# all dirs
+KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
+KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
+export KBUILD_ALL_DIRS
+
 # ===========================================================================
 # Rules shared between *config targets and build targets


And then use this definition in all find / tar operations.
We do not need to change everything in one go as the old RCS_TAR_IGNORE
and RCS_FIND_IGNORE can continue as is for a while.

Can you please try out this approach?

	Sam

Re: packaging issues: some serious, some not so serious

[Robert P. J. Day]

On Mon, 18 Feb 2008, Sam Ravnborg wrote:

> I would much rather you persuaded your earlier idea list all the
> directories to visit. This list is known and easy to maintain
> whereas a blacklist like we have now only will grow bigger over
> time.
>
> We could add something like:
> diff --git a/Makefile b/Makefile
> index 0d585c0..fb5bbc3 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
>  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
>  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
>
> +# all dirs
> +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> +export KBUILD_ALL_DIRS
> +
>  # ===========================================================================
>  # Rules shared between *config targets and build targets
>
>
> And then use this definition in all find / tar operations.
> We do not need to change everything in one go as the old RCS_TAR_IGNORE
> and RCS_FIND_IGNORE can continue as is for a while.
>
> Can you please try out this approach?

sure, i prefer that strategy anyway.  let me work on it.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: packaging issues: some serious, some not so serious

[Robert P. J. Day]

On Mon, 18 Feb 2008, Sam Ravnborg wrote:

> We could add something like:
> diff --git a/Makefile b/Makefile
> index 0d585c0..fb5bbc3 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
>  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
>  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
>
> +# all dirs
> +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> +export KBUILD_ALL_DIRS
> +

would you not also need to include the regular files at the top of the
source tree as well?  for some of the packaging targets, those files
should be included as well.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: packaging issues: some serious, some not so serious

[Sam Ravnborg]

On Tue, Feb 19, 2008 at 04:43:56AM -0500, Robert P. J. Day wrote:
> On Mon, 18 Feb 2008, Sam Ravnborg wrote:
> 
> > We could add something like:
> > diff --git a/Makefile b/Makefile
> > index 0d585c0..fb5bbc3 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
> >  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
> >  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
> >
> > +# all dirs
> > +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> > +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> > +export KBUILD_ALL_DIRS
> > +
> 
> would you not also need to include the regular files at the top of the
> source tree as well?  for some of the packaging targets, those files
> should be included as well.
Yes.
'find --max-depth=1 *' or maybe we should provide a seperate list
for all the relevant filenmaes?

	Sam

Re: packaging issues: some serious, some not so serious

[Robert P. J. Day]

On Tue, 19 Feb 2008, Sam Ravnborg wrote:

> On Tue, Feb 19, 2008 at 04:43:56AM -0500, Robert P. J. Day wrote:
> > On Mon, 18 Feb 2008, Sam Ravnborg wrote:
> >
> > > We could add something like:
> > > diff --git a/Makefile b/Makefile
> > > index 0d585c0..fb5bbc3 100644
> > > --- a/Makefile
> > > +++ b/Makefile
> > > @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
> > >  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
> > >  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
> > >
> > > +# all dirs
> > > +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> > > +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> > > +export KBUILD_ALL_DIRS
> > > +
> >
> > would you not also need to include the regular files at the top of the
> > source tree as well?  for some of the packaging targets, those files
> > should be included as well.
> Yes.
> 'find --max-depth=1 *' or maybe we should provide a seperate list
> for all the relevant filenmaes?

oh, dang ... it just occurred to me that it's still a good idea to add
dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE
variables since someone might, weirdly, decide to mount a subdirectory
of the kernel source tree on a separate partition.  unlikely, yes, but
it can happen.

it may be that we want a more general variable for those blacklists as
well -- ALL_FIND_IGNORE, ALL_TAR_IGNORE, that sort of thing.

thoughts?

rday
--


========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: packaging issues: some serious, some not so serious

[Sam Ravnborg]

On Tue, Feb 19, 2008 at 05:15:10AM -0500, Robert P. J. Day wrote:
> On Tue, 19 Feb 2008, Sam Ravnborg wrote:
> 
> > On Tue, Feb 19, 2008 at 04:43:56AM -0500, Robert P. J. Day wrote:
> > > On Mon, 18 Feb 2008, Sam Ravnborg wrote:
> > >
> > > > We could add something like:
> > > > diff --git a/Makefile b/Makefile
> > > > index 0d585c0..fb5bbc3 100644
> > > > --- a/Makefile
> > > > +++ b/Makefile
> > > > @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
> > > >  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
> > > >  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
> > > >
> > > > +# all dirs
> > > > +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> > > > +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> > > > +export KBUILD_ALL_DIRS
> > > > +
> > >
> > > would you not also need to include the regular files at the top of the
> > > source tree as well?  for some of the packaging targets, those files
> > > should be included as well.
> > Yes.
> > 'find --max-depth=1 *' or maybe we should provide a seperate list
> > for all the relevant filenmaes?
> 
> oh, dang ... it just occurred to me that it's still a good idea to add
> dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE
> variables since someone might, weirdly, decide to mount a subdirectory
> of the kernel source tree on a separate partition.  unlikely, yes, but
> it can happen.

If they do I would not care at all.
We should strive to make the normal cases behave as expected and try
to come up with general solutions that does not prevent the more
exotic cases.  But trying to imagine all the weird cases and handle
them will only result in an un-maintainable mess.

	Sam

Re: packaging issues: some serious, some not so serious

[Robert P. J. Day]

On Tue, 19 Feb 2008, Sam Ravnborg wrote:

> On Tue, Feb 19, 2008 at 05:15:10AM -0500, Robert P. J. Day wrote:
> > On Tue, 19 Feb 2008, Sam Ravnborg wrote:
> >
> > > On Tue, Feb 19, 2008 at 04:43:56AM -0500, Robert P. J. Day wrote:
> > > > On Mon, 18 Feb 2008, Sam Ravnborg wrote:
> > > >
> > > > > We could add something like:
> > > > > diff --git a/Makefile b/Makefile
> > > > > index 0d585c0..fb5bbc3 100644
> > > > > --- a/Makefile
> > > > > +++ b/Makefile
> > > > > @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
> > > > >  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
> > > > >  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
> > > > >
> > > > > +# all dirs
> > > > > +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> > > > > +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> > > > > +export KBUILD_ALL_DIRS
> > > > > +
> > > >
> > > > would you not also need to include the regular files at the top of the
> > > > source tree as well?  for some of the packaging targets, those files
> > > > should be included as well.
> > > Yes.
> > > 'find --max-depth=1 *' or maybe we should provide a seperate list
> > > for all the relevant filenmaes?
> >
> > oh, dang ... it just occurred to me that it's still a good idea to
> > add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE
> > variables since someone might, weirdly, decide to mount a
> > subdirectory of the kernel source tree on a separate partition.
> > unlikely, yes, but it can happen.
>
> If they do I would not care at all. We should strive to make the
> normal cases behave as expected and try to come up with general
> solutions that does not prevent the more exotic cases.  But trying
> to imagine all the weird cases and handle them will only result in
> an un-maintainable mess.

  i agree.  from what seemed like a simple fix, this has gotten a bit
out of hand.  i think the easiest solution would be to just come up
with a single whitelist, and a number of command-specific blacklists,
and leave it at that.  that's guaranteed to work, and it's trivially
extensible.

  so, as an example, define the global whitelist:

KBUILD_ALL_FILES := COPYING CREDITS Kbuild MAINTAINERS Makefile README REPORTING-BUGS
KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init ipc
KBUILD_ALL_DIRS += kernel lib mm net samples scripts security sound usr virt
KBUILD_ALL SRCS := $(KBUILD_ALL_FILES) $(KBUILD_ALL_DIRS)
export KBUILD_ALL_FILES KBUILD_ALL_DIRS KBUILD_ALL_SRCS

those variables above are valid for everything and are easily
augmented.

  in terms of what are currently called RCS_FIND_IGNORE and
RCS_TAR_IGNORE, scrap those entirely as they clearly reflect only
"rcs"-related objects, and just invent new variables:

  FIND_OBJS_TO_IGNORE := ... both files and dirs ...
  TAR_OBJS_TO_IGNORE := ... both files and dirs ...

and that's it.  problem solved.  don't try to partition off the
RCS-related stuff into its own list of objects, it's not worth the
trouble.  and note how that entirely solves the lost+found issue, and
any other issues that might ever arise.

  thoughts?

rday
--



========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: packaging issues: some serious, some not so serious

[Sam Ravnborg]

On Tue, Feb 19, 2008 at 07:25:48AM -0500, Robert P. J. Day wrote:
> On Tue, 19 Feb 2008, Sam Ravnborg wrote:
> 
> > On Tue, Feb 19, 2008 at 05:15:10AM -0500, Robert P. J. Day wrote:
> > > On Tue, 19 Feb 2008, Sam Ravnborg wrote:
> > >
> > > > On Tue, Feb 19, 2008 at 04:43:56AM -0500, Robert P. J. Day wrote:
> > > > > On Mon, 18 Feb 2008, Sam Ravnborg wrote:
> > > > >
> > > > > > We could add something like:
> > > > > > diff --git a/Makefile b/Makefile
> > > > > > index 0d585c0..fb5bbc3 100644
> > > > > > --- a/Makefile
> > > > > > +++ b/Makefile
> > > > > > @@ -358,6 +358,11 @@ export MODVERDIR := $(if $(KBUILD_EXTMOD),$(firstword $(KBU
> > > > > >  RCS_FIND_IGNORE := \( -name SCCS -o -name BitKeeper -o -name .svn -o -name CVS
> > > > > >  export RCS_TAR_IGNORE := --exclude SCCS --exclude BitKeeper --exclude .svn --ex
> > > > > >
> > > > > > +# all dirs
> > > > > > +KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init
> > > > > > +KBUILD_ALL_DIRS += ipc net samples scripts security sound usr virt
> > > > > > +export KBUILD_ALL_DIRS
> > > > > > +
> > > > >
> > > > > would you not also need to include the regular files at the top of the
> > > > > source tree as well?  for some of the packaging targets, those files
> > > > > should be included as well.
> > > > Yes.
> > > > 'find --max-depth=1 *' or maybe we should provide a seperate list
> > > > for all the relevant filenmaes?
> > >
> > > oh, dang ... it just occurred to me that it's still a good idea to
> > > add dirs like lost+found to the RCS_FIND_IGNORE and RCS_TAR_IGNORE
> > > variables since someone might, weirdly, decide to mount a
> > > subdirectory of the kernel source tree on a separate partition.
> > > unlikely, yes, but it can happen.
> >
> > If they do I would not care at all. We should strive to make the
> > normal cases behave as expected and try to come up with general
> > solutions that does not prevent the more exotic cases.  But trying
> > to imagine all the weird cases and handle them will only result in
> > an un-maintainable mess.
> 
>   i agree.  from what seemed like a simple fix, this has gotten a bit
> out of hand.  i think the easiest solution would be to just come up
> with a single whitelist, and a number of command-specific blacklists,
> and leave it at that.  that's guaranteed to work, and it's trivially
> extensible.
> 
>   so, as an example, define the global whitelist:
> 
> KBUILD_ALL_FILES := COPYING CREDITS Kbuild MAINTAINERS Makefile README REPORTING-BUGS
> KBUILD_ALL_DIRS := arch block crypto Documentation drivers fs include init ipc
> KBUILD_ALL_DIRS += kernel lib mm net samples scripts security sound usr virt
> KBUILD_ALL SRCS := $(KBUILD_ALL_FILES) $(KBUILD_ALL_DIRS)
> export KBUILD_ALL_FILES KBUILD_ALL_DIRS KBUILD_ALL_SRCS
> 
> those variables above are valid for everything and are easily
> augmented.
> 
>   in terms of what are currently called RCS_FIND_IGNORE and
> RCS_TAR_IGNORE, scrap those entirely as they clearly reflect only
> "rcs"-related objects, and just invent new variables:
> 
>   FIND_OBJS_TO_IGNORE := ... both files and dirs ...
>   TAR_OBJS_TO_IGNORE := ... both files and dirs ...
> 
> and that's it.  problem solved.  don't try to partition off the
> RCS-related stuff into its own list of objects, it's not worth the
> trouble.  and note how that entirely solves the lost+found issue, and
> any other issues that might ever arise.
> 
>   thoughts?

Looks good - more or less in line with how I think we should do it.
I assume you cook up a patch for this. Try one of the simple cases first.

	Sam