2.5.21: kbuild changes broke filenames with commas

2.5.21: kbuild changes broke filenames with commas

[Russell King]

With the latest kbuild version in 2.5.21, we are unable to build the
following files:

linux/drivers/block/smart1,2.h
linux/drivers/scsi/53c7,8xx.c
linux/drivers/scsi/53c7,8xx.h
linux/drivers/scsi/53c7,8xx.scr
linux/arch/arm/mm/proc-arm6,7.S
linux/arch/arm/mm/proc-arm2,3.S

This is because we end up passing gcc the following argument:

	-Wp,-MD,.proc-arm6,7.o.d

which gets passed to cpp0 as:

	-MD .proc-arm6 7.o.d
	              ^ space, not comma

and therefore cpp0 sees "-MD", ".proc-arm6" and "7.o.d" as separate
arguments.

There seems to be two solutions:

1. renaming all the above files to contain '_' instead of ','.
2. see if kbuild can use the DEPENDENCIES_OUTPUT environment variable

Kai pointed out that we've already got one exception in kbuild to fixup
the filename for KBUILD_BASENAME (, -> _ and that's not a weird smilie!)
so (1) is probably going to be better, and we can get rid of the special
"comma" handling.

Either way, I plan to rename the two ARM files.  That leaves the 53c7,8xx
driver and that block header file.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Upper limits to number of processes / threads in Linux / ia32??

[M. Edward (Ed) Borasky]

Are there any practical limits in Linux on the ia32 to the number of processes /
threads that can be managed at one time? What's the largest number of
threads that anyone has seen work?
--
How to Stop A Folksinger Cold # 2
"Are you going to Scarborough Fair?..."
No.

Re: 2.5.21: kbuild changes broke filenames with commas

[Kai Germaschewski]

On Sun, 9 Jun 2002, Russell King wrote:

> With the latest kbuild version in 2.5.21, we are unable to build the
> following files:
> 
> linux/drivers/block/smart1,2.h
> linux/drivers/scsi/53c7,8xx.c
> linux/drivers/scsi/53c7,8xx.h
> linux/drivers/scsi/53c7,8xx.scr
> linux/arch/arm/mm/proc-arm6,7.S
> linux/arch/arm/mm/proc-arm2,3.S

Well, we're unable to build drivers/scsi/53c7,8xx.c and the two ARM files,
the others just get included, so don't affect the build. 53c7,8xxx is 
broken by BIO changes, so nobody but ARM is affected currently.

> This is because we end up passing gcc the following argument:
> 
> 	-Wp,-MD,.proc-arm6,7.o.d
> 
> which gets passed to cpp0 as:
> 
> 	-MD .proc-arm6 7.o.d
> 	              ^ space, not comma
> 
> and therefore cpp0 sees "-MD", ".proc-arm6" and "7.o.d" as separate
> arguments.
> 
> There seems to be two solutions:
> 
> 1. renaming all the above files to contain '_' instead of ','.
> 2. see if kbuild can use the DEPENDENCIES_OUTPUT environment variable

There's three way around the limitation of "-Wp" to pass arguments 
containing commas to cpp:
o Use the environment variable SUNPRO_DEPENDENCIES instead or
o Invoke cpp directly or
o Rename the generated temporary .d file by substituting ',' with
  something else.

> Kai pointed out that we've already got one exception in kbuild to fixup
> the filename for KBUILD_BASENAME (, -> _ and that's not a weird smilie!)
> so (1) is probably going to be better, and we can get rid of the special
> "comma" handling.

It would in general be saner to generally disallow source file names with 
a ',' in them. Not so much for the build system, but also for the other 
cases where we can't handle them, like e.g. KBUILD_BASENAME as used in 
include/linux/spinlock.h, or in Rusty's work, which will allow to pass
parameters to built-in modules on the kernel command line, like
<module name>.<parm>=<value>.

--Kai

Re: 2.5.21: kbuild changes broke filenames with commas

[Tom Rini]

On Sun, Jun 09, 2002 at 05:58:04PM +0100, Russell King wrote:

> With the latest kbuild version in 2.5.21, we are unable to build the
> following files:
> 
[snip]
> linux/drivers/scsi/53c7,8xx.c
> linux/drivers/scsi/53c7,8xx.h
> linux/drivers/scsi/53c7,8xx.scr

How about we remove these alltogether?  The ncr53c8xx and sym-2 drivers
both support the 53c810, 53c825, and 53c820 devices and there's a 'common'
53c7xx backend which handles the 53c700, 53x700-66 53c710 and 53c720
chipsets that the 53c7,8xx driver handles.  The only minor issue is that
if there's any ISA (PCI?) cards with these chipsets would need a new
front-end to detect the cards.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: 2.5.21: kbuild changes broke filenames with commas

[Keith Owens]

On Sun, 9 Jun 2002 17:58:04 +0100, 
Russell King <rmk@arm.linux.org.uk> wrote:
>With the latest kbuild version in 2.5.21, we are unable to build the
>following files:
>
>linux/drivers/block/smart1,2.h
>linux/drivers/scsi/53c7,8xx.c
>linux/drivers/scsi/53c7,8xx.h
>linux/drivers/scsi/53c7,8xx.scr
>linux/arch/arm/mm/proc-arm6,7.S
>linux/arch/arm/mm/proc-arm2,3.S

kbuild 2.5 can handle filenames with ',' in the name.  I do not believe
in restricting what users can do unless there is absolutely no
alternative.  In this case a smarter build system can handle special
filenames.

Re: 2.5.21: kbuild changes broke filenames with commas

[Russell King]

On Tue, Jun 11, 2002 at 09:02:45AM +1000, Keith Owens wrote:
> >linux/drivers/block/smart1,2.h
> >linux/drivers/scsi/53c7,8xx.c
> >linux/drivers/scsi/53c7,8xx.h
> >linux/drivers/scsi/53c7,8xx.scr
> >linux/arch/arm/mm/proc-arm6,7.S
> >linux/arch/arm/mm/proc-arm2,3.S
> 
> kbuild 2.5 can handle filenames with ',' in the name.  I do not believe
> in restricting what users can do unless there is absolutely no
> alternative.  In this case a smarter build system can handle special
> filenames.

I've already fixed up the two ARM ones.  That leaves one problematic
file - 53c7,8xx.c.

Is it really worth adding complexity to a build system to work around
what is really a GCC bug for just one file?  I don't think so.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: 2.5.21: kbuild changes broke filenames with commas

[Thunder from the hill]

Hi,

On Tue, 11 Jun 2002, Russell King wrote:
> Is it really worth adding complexity to a build system to work around
> what is really a GCC bug for just one file?  I don't think so.

Think big! Think about __future__.

Regards,
Thunder
-- 
German attitude becoming        |	Thunder from the hill at ngforever
rightaway popular:		|
       "Get outa my way,  	|	free inhabitant not directly
    for I got a mobile phone!"	|	belonging anywhere

Re: 2.5.21: kbuild changes broke filenames with commas

[Russell King]

On Tue, Jun 11, 2002 at 05:11:13AM -0600, Thunder from the hill wrote:
> On Tue, 11 Jun 2002, Russell King wrote:
> > Is it really worth adding complexity to a build system to work around
> > what is really a GCC bug for just one file?  I don't think so.
> 
> Think big! Think about __future__.

The path to overdesign and excessive featurebloat.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: 2.5.21: kbuild changes broke filenames with commas

[Thunder from the hill]

Hi

On Tue 11 Jun 2002 Russell King wrote:
> On Tue Jun 11 2002 at 05:11:13AM -0600 Thunder from the hill wrote:
> > Think big! Think about __future__.
> 
> The path to overdesign and excessive featurebloat.

Well the question is then how will things look without commas? I suppose 
if we have very complex things and prevent using commas whereas I don't 
assert that we do for text but this is just a bloat example it's good to 
have things like commas allowed even though we are in case we won't allow 
them there talking about file names.

If we allow commas all over the filesystem and likewise say that there is 
nothing to mention about it why should we refuse them for kbuild 
especially since there is a parallel system which allows commas?

Regards
Thunder
-- 
German attitude becoming        |	Thunder from the hill at ngforever
rightaway popular:		|
       "Get outa my way  	|	free inhabitant not directly
    for I got a mobile phone!"	|	belonging anywhere

Re: 2.5.21: kbuild changes broke filenames with commas

[Russell King]

On Tue, Jun 11, 2002 at 06:16:10AM -0600, Thunder from the hill wrote:
> Well the question is then how will things look without commas? I suppose 
> if we have very complex things and prevent using commas whereas I don't 
> assert that we do for text but this is just a bloat example it's good to 
> have things like commas allowed even though we are in case we won't allow 
> them there talking about file names.
> 
> If we allow commas all over the filesystem and likewise say that there is 
> nothing to mention about it why should we refuse them for kbuild 
> especially since there is a parallel system which allows commas?

You've *completely* missed the point.

The gcc argument >>> -Wp,-MD,foo,bar.c <<< is the problem.  If anything
should be fixed, its that silly gcc syntax.  kbuild should not work
around the inability of gcc to accept filenames with commas in.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: 2.5.21: kbuild changes broke filenames with commas

[Ingo Oeser]

On Tue, Jun 11, 2002 at 06:16:10AM -0600, Thunder from the hill wrote:
> If we allow commas all over the filesystem and likewise say that there is 
> nothing to mention about it why should we refuse them for kbuild 
> especially since there is a parallel system which allows commas?

Aehm, if you argument with what we allow in the filesystem, then
I could argue, that everything except '\0' and '/' is allowed
to build paths.

Interesting is what the C-Standard says about naming conventions
for files.

The rest is merely taste and limitations of the build system.

I like ',' as much as ' ' and quotes in file names. But sh*t
happens and we usally have to cope with it :-/

Regards

Ingo Oeser
-- 
Science is what we can tell a computer. Art is everything else. --- D.E.Knuth

Re: 2.5.21: kbuild changes broke filenames with commas

[Thunder from the hill]

Hi,

On Wed, 12 Jun 2002, Ingo Oeser wrote:
> On Tue, Jun 11, 2002 at 06:16:10AM -0600, Thunder from the hill wrote:
> > If we allow commas all over the filesystem and likewise say that there is 
> > nothing to mention about it why should we refuse them for kbuild 
> > especially since there is a parallel system which allows commas?
> 
> Aehm, if you argument with what we allow in the filesystem

I just said that there seems a way to do it.

Regards,
Thunder
-- 
German attitude becoming        |	Thunder from the hill at ngforever
rightaway popular:		|
       "Get outa my way,  	|	free inhabitant not directly
    for I got a mobile phone!"	|	belonging anywhere

Re: 2.5.21: kbuild changes broke filenames with commas

[Thunder from the hill]

Hi,

On Tue, 11 Jun 2002, Russell King wrote:
> > Think big! Think about __future__.
> 
> The path to overdesign and excessive featurebloat.

Hmmm... You can overdo both sides. OTOH, why do we need IrDA drivers? 
Featurebloat? Or new processor impls? They're certainly features, required 
to run the kernel under certain (very limited) conditions. I don't think 
you would remove them!

Regards,
Thunder
-- 
German attitude becoming        |	Thunder from the hill at ngforever
rightaway popular:		|
       "Get outa my way,  	|	free inhabitant not directly
    for I got a mobile phone!"	|	belonging anywhere

Re: 2.5.21: kbuild changes broke filenames with commas

[Daniel Phillips]

On Tuesday 11 June 2002 09:39, Russell King wrote:
> On Tue, Jun 11, 2002 at 09:02:45AM +1000, Keith Owens wrote:
> > >linux/drivers/block/smart1,2.h
> > >linux/drivers/scsi/53c7,8xx.c
> > >linux/drivers/scsi/53c7,8xx.h
> > >linux/drivers/scsi/53c7,8xx.scr
> > >linux/arch/arm/mm/proc-arm6,7.S
> > >linux/arch/arm/mm/proc-arm2,3.S
> > 
> > kbuild 2.5 can handle filenames with ',' in the name.  I do not believe
> > in restricting what users can do unless there is absolutely no
> > alternative.  In this case a smarter build system can handle special
> > filenames.
> 
> I've already fixed up the two ARM ones.  That leaves one problematic
> file - 53c7,8xx.c.
> 
> Is it really worth adding complexity to a build system to work around
> what is really a GCC bug for just one file?  I don't think so.

Are you sure that complexity was added just to handle commas in names?
Or is it really an example of how good design never gave this bug a
chance to exist in the first palce.

I *really* don't like the idea of papering over such bugs by curing the
symptoms, as you seem to be advocating.

-- 
Daniel

Re: 2.5.21: kbuild changes broke filenames with commas

[Alexander Viro]

On Tue, 11 Jun 2002, Daniel Phillips wrote:

> > Is it really worth adding complexity to a build system to work around
> > what is really a GCC bug for just one file?  I don't think so.
> 
> Are you sure that complexity was added just to handle commas in names?
> Or is it really an example of how good design never gave this bug a
> chance to exist in the first palce.
> 
> I *really* don't like the idea of papering over such bugs by curing the
> symptoms, as you seem to be advocating.

AFAICS you are advocating exactly that - papering over the idiotic
gcc options syntax by kludges in build system...

Re: 2.5.21: kbuild changes broke filenames with commas

[Daniel Phillips]

On Tuesday 11 June 2002 18:14, Alexander Viro wrote:
> On Tue, 11 Jun 2002, Daniel Phillips wrote:
> 
> > > Is it really worth adding complexity to a build system to work around
> > > what is really a GCC bug for just one file?  I don't think so.
> > 
> > Are you sure that complexity was added just to handle commas in names?
> > Or is it really an example of how good design never gave this bug a
> > chance to exist in the first palce.
> > 
> > I *really* don't like the idea of papering over such bugs by curing the
> > symptoms, as you seem to be advocating.
> 
> AFAICS you are advocating exactly that - papering over the idiotic
> gcc options syntax by kludges in build system...

So you want to leave this hole sitting there waiting for someone else
to step in it, until gcc gets fixed?  I'd think a bug report would be
a more effective solution than holding one's breath until blue in the
face.

I don't have a problem with fixing this at the interface to gcc level,
as then it's fixed definitively for the whole kbuild system.  I do have
a problem with just changing the names of files that trigger the bug,
that's sloppy beyond the belief and it's the stuff that substandard
systems are made of.

-- 
Daniel

Re: 2.5.21: kbuild changes broke filenames with commas

[Russell King]

On Tue, Jun 11, 2002 at 06:08:22PM +0200, Daniel Phillips wrote:
> Are you sure that complexity was added just to handle commas in names?
> Or is it really an example of how good design never gave this bug a
> chance to exist in the first palce.
> 
> I *really* don't like the idea of papering over such bugs by curing the
> symptoms, as you seem to be advocating.

Lets see.

We have two places in 2.5.21 where commas are unacceptable:

1. -DKBUILD_BASENAME=$(subst $(comma),_,$(subst -,_,$(*F)))

   We need to do this because to do because KBUILD_BASENAME is used in
   places where commas are not acceptable.

   We papered over the fact that make can't subst commas by using the
   $(comma) construct.

2. -Wp,-MD,.$(subst /,_,$@).d (currently unfixed)

   This would need to become something like:

   -Wp,-MD,.$(subst /,_,$(subst $(comma,_,$@)).d

So now we have two places where the same yucky substing of commas to
something more palettable happens.  Now, what if we had:

	foo,bar.c

and

	foo_bar.c

in the same directory?  The kbuild system goes wrong, destroying dependency
information, using the wrong KBUILD_BASENAME.  Oops.  I guess we papered
over a bug by allowing commas in filenames.

In addition, I'd like to point out the following paragraph in the make
info files:

   The variant variables' names are formed by
   appending `D' or `F', respectively.  These variants are semi-obsolete
   in GNU `make' since the functions `dir' and `notdir' can be used to get
   a similar effect (*note Functions for File Names: File Name
   Functions.).

Both kbuild-2.5 and the existing kernel build make heavy use of the
"$(*F)" notation.  Should we really be putting semi-obsolete features
into either of the kernel build system?

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: 2.5.21: kbuild changes broke filenames with commas

[Daniel Phillips]

On Tuesday 11 June 2002 18:31, Russell King wrote:
> On Tue, Jun 11, 2002 at 06:08:22PM +0200, Daniel Phillips wrote:
> > Are you sure that complexity was added just to handle commas in names?
> > Or is it really an example of how good design never gave this bug a
> > chance to exist in the first palce.
> > 
> > I *really* don't like the idea of papering over such bugs by curing the
> > symptoms, as you seem to be advocating.
> 
> Lets see.
> 
> We have two places in 2.5.21 where commas are unacceptable:
> 
> 1. -DKBUILD_BASENAME=$(subst $(comma),_,$(subst -,_,$(*F)))
> 
>    We need to do this because to do because KBUILD_BASENAME is used in
>    places where commas are not acceptable.
> 
>    We papered over the fact that make can't subst commas by using the
>    $(comma) construct.
> 
> 2. -Wp,-MD,.$(subst /,_,$@).d (currently unfixed)
> 
>    This would need to become something like:
> 
>    -Wp,-MD,.$(subst /,_,$(subst $(comma,_,$@)).d
> 
> So now we have two places where the same yucky substing of commas to
> something more palettable happens.  Now, what if we had:
> 
> 	foo,bar.c
> 
> and
> 
> 	foo_bar.c
> 
> in the same directory?  The kbuild system goes wrong, destroying dependency
> information, using the wrong KBUILD_BASENAME.  Oops.  I guess we papered
> over a bug by allowing commas in filenames.

Yup.

> In addition, I'd like to point out the following paragraph in the make
> info files:
> 
>    The variant variables' names are formed by
>    appending `D' or `F', respectively.  These variants are semi-obsolete
>    in GNU `make' since the functions `dir' and `notdir' can be used to get
>    a similar effect (*note Functions for File Names: File Name
>    Functions.).
> 
> Both kbuild-2.5 and the existing kernel build make heavy use of the
> "$(*F)" notation.  Should we really be putting semi-obsolete features
> into either of the kernel build system?

No, agreed.  Now this is sensible.

-- 
Daniel

Re: 2.5.21: kbuild changes broke filenames with commas

[Keith Owens]

On Tuesday 11 June 2002 18:31, Russell King wrote:
> On Tue, Jun 11, 2002 at 06:08:22PM +0200, Daniel Phillips wrote:
> So now we have two places where the same yucky substing of commas to
> something more palettable happens.

So what?  Users want filenames with ',' in them, the build system
should cope with it.  Restricting what the user is allowed to do to
what the build system can handle is the wrong approach.  The build
system already has to replace '-' with '_', changing comma as well is
not a problem.  Or are you going to say that '-' is not allowed in
filenames either?

>Now, what if we had:
> 
> 	foo,bar.c
> 
> and
> 
> 	foo_bar.c
> 
> in the same directory?  The kbuild system goes wrong, destroying dependency
> information, using the wrong KBUILD_BASENAME.  Oops.  I guess we papered
> over a bug by allowing commas in filenames.

Not in kbuild 2.5.  I handle this case correctly for the -MD dependency
filename.  Try it and see.

KBUILD_BASENAME and KBUILD_OBJECTNAME could suffer from name
collisions.  BASENAME is only used inside objects so duplicate
basenames are not a problem.

OBJECTNAME is externally visible, it is used in Rusty's rationalization
of boot and module parameters.  The only time that OBJECTNAME collision
would be a problem is when there are two modules called foo,bar and
foo_bar.  Having two modules that differ by a single character in the
middle of the name is going to cause more problems than just option
collision.  BTW, the existing build system does not support
KBUILD_OBJECTNAME so Rusty's code cannot go in.

> Both kbuild-2.5 and the existing kernel build make heavy use of the
> "$(*F)" notation.  Should we really be putting semi-obsolete features
> into either of the kernel build system?

That would be worth changing, it is just a textual change with no
effect on the code.

Re: 2.5.21: kbuild changes broke filenames with commas

[Kai Germaschewski]

On Wed, 12 Jun 2002, Keith Owens wrote:

> So what?  Users want filenames with ',' in them, the build system
> should cope with it.  Restricting what the user is allowed to do to
> what the build system can handle is the wrong approach.  The build
> system already has to replace '-' with '_', changing comma as well is
> not a problem.  Or are you going to say that '-' is not allowed in
> filenames either?

It's a stupid discussion - I added support for filenames containing a ',', 
but the only remaining user is 53c7,8xx.c. That one is broken by the BIO 
changes anyway, and I heard people say it should go away, as the 
hardware is supported by other drivers. So I'll wait and see, if it 
doesn't get fixed but removed, I think removing the hacks to support ',' 
in the filename is the way to go.

> >Now, what if we had:
> > 
> > 	foo,bar.c
> > 
> > and
> > 
> > 	foo_bar.c
> > 
> > in the same directory?  The kbuild system goes wrong, destroying dependency
> > information, using the wrong KBUILD_BASENAME.  Oops.  I guess we papered
> > over a bug by allowing commas in filenames.
> 
> Not in kbuild 2.5.  I handle this case correctly for the -MD dependency
> filename.  Try it and see.

Well, let me rephrase it as "foo,bar.c" and "foo:bar.c" ;). kbuild-2.5
would break. Of course it's fixable with even more workarounds, but that's
not the point.

> OBJECTNAME is externally visible, it is used in Rusty's rationalization
> of boot and module parameters.  The only time that OBJECTNAME collision
> would be a problem is when there are two modules called foo,bar and
> foo_bar.  Having two modules that differ by a single character in the
> middle of the name is going to cause more problems than just option
> collision.  BTW, the existing build system does not support
> KBUILD_OBJECTNAME so Rusty's code cannot go in.

Rusty knows that the current build system can support KBUILD_OBJECT (which 
is what you called it, not KBUILD_OBJECTNAME) just fine - it's a three 
line diff, but I don't see a point in submitting it as long as nobody uses 
it.

--Kai