Strange file names

Strange file names

[Gary Thomas]

I just did a pull on poky master and now have a lot of files
with strange names like these:
   poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto-rt_3.0": }} linux-yocto-rt_3.0.bbappend
   poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto-rt_3.2": }} linux-yocto-rt_3.2.bbappend
   poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto_3.0": }} linux-yocto_3.0.bbappend
   poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto_3.2": }} linux-yocto_3.2.bbappend

Is this right?  What are they all about?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Strange file names

[Tom Zanussi]

On Tue, 2012-03-27 at 08:40 -0600, Gary Thomas wrote:
> I just did a pull on poky master and now have a lot of files
> with strange names like these:
>    poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto-rt_3.0": }} linux-yocto-rt_3.0.bbappend
>    poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto-rt_3.2": }} linux-yocto-rt_3.2.bbappend
>    poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto_3.0": }} linux-yocto_3.0.bbappend
>    poky/scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/{{ if kernel_choice == "linux-yocto_3.2": }} linux-yocto_3.2.bbappend
> 
> Is this right?  What are they all about?
> 

These are part of the new Yocto BSP Tools (yocto-bsp) and all the files
under 'scripts/lib/bsp/substrate' are template files for the tool.

The template files can contain inline Python between {{ tags }} that
gets used as logic when generating the BSP.  The inline Python can also
be used in the filne names themselves for that purpose.  In this case,
it's used to only include one of the .bbappend files depending on the
value of the kernel_choice variable.

It may seem strange, but filenames are just text after all, and I wanted
to avoid having separate 'meta-files' describing that same logic that
would also have to be kept in sync, so decided it would be less
error-prone to keep that information in the filenames themselves.

I'm in the process of writing all this up to put in the BSP manual, but
not quite there yet...

Tom

Re: Strange file names

[Wolfgang Denk]

Dear Tom Zanussi,

In message <1332859973.3854.15.camel@elmorro> you wrote:
>
> The template files can contain inline Python between {{ tags }} that
> gets used as logic when generating the BSP.  The inline Python can also
> be used in the filne names themselves for that purpose.  In this case,
> it's used to only include one of the .bbappend files depending on the
> value of the kernel_choice variable.

Arghhh... This is totally broken. Do not do this.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Successful and fortunate crime is called virtue.             - Seneca

Re: Strange file names

[Tom Zanussi]

On Tue, 2012-03-27 at 19:43 +0200, Wolfgang Denk wrote:
> Dear Tom Zanussi,
> 
> In message <1332859973.3854.15.camel@elmorro> you wrote:
> >
> > The template files can contain inline Python between {{ tags }} that
> > gets used as logic when generating the BSP.  The inline Python can also
> > be used in the filne names themselves for that purpose.  In this case,
> > it's used to only include one of the .bbappend files depending on the
> > value of the kernel_choice variable.
> 
> Arghhh... This is totally broken. Do not do this.
> 

Why do you say so - they're legal filenames and useful for this purpose?

Tom

> Best regards,
> 
> Wolfgang Denk
> 

Re: Strange file names

[Wolfgang Denk]

Dear Tom Zanussi,

In message <1332870781.20140.12.camel@elmorro> you wrote:
>
> > Arghhh... This is totally broken. Do not do this.
> 
> Why do you say so - they're legal filenames and useful for this purpose?

Not anything that is legal (in the sense that it stretches the
technical possibilities to the limit) actually makes sense.

Yes, Unix allows nearly arbitrary file names.  Yes, it is technically
possible to store the source code of a program or a script in a file
name.  The fact that nobody does this should make you think if this is
a clever approach that nobody else could figure out, or if it is just
plain ......

Please consider the mental welfare of the people who have to maintain
that code after you, who have to type file names like "{{ if
kernel_choice == "linux-yocto-rt_3.2": }} linux-yocto-rt_3.2.bbappend".

Even standard commands like "ls" produce output that make my toe nails
roll up:

$ ls scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/
files
{{ if kernel_choice == "linux-yocto_3.0": }} linux-yocto_3.0.bbappend
{{ if kernel_choice == "linux-yocto_3.2": }} linux-yocto_3.2.bbappend
{{ if kernel_choice == "linux-yocto-rt_3.0": }} linux-yocto-rt_3.0.bbappend
{{ if kernel_choice == "linux-yocto-rt_3.2": }} linux-yocto-rt_3.2.bbappend
kernel-list.noinstall
$

PLEASE do not to this!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
History is only a confused heap of facts.
                                       -- Philip Earl of Chesterfield

Re: Strange file names

[Tom Zanussi]

On Tue, 2012-03-27 at 20:22 +0200, Wolfgang Denk wrote:
> Dear Tom Zanussi,
> 
> In message <1332870781.20140.12.camel@elmorro> you wrote:
> >
> > > Arghhh... This is totally broken. Do not do this.
> > 
> > Why do you say so - they're legal filenames and useful for this purpose?
> 
> Not anything that is legal (in the sense that it stretches the
> technical possibilities to the limit) actually makes sense.
> 
> Yes, Unix allows nearly arbitrary file names.  Yes, it is technically
> possible to store the source code of a program or a script in a file
> name.  The fact that nobody does this should make you think if this is
> a clever approach that nobody else could figure out, or if it is just
> plain ......
> 

Yeah, as I mentioned, the motivation was that it seemed the most
straightforward thing to do, notwithstanding the ugliness of the file
names.

> Please consider the mental welfare of the people who have to maintain
> that code after you, who have to type file names like "{{ if
> kernel_choice == "linux-yocto-rt_3.2": }} linux-yocto-rt_3.2.bbappend".
> 

Believe me, I did consider it, since I was the one who had to time them
in in the first place. ;-).

And I have to agree, it is a pain to edit them, mainly because of the
need for quotes - actually I finally resorted to using Nautilus which
made it much easier.

In the end, I balanced that against the fact that it would be very few
people editing these files very rarely (once a release basically), and
decided to do it anyway.

> Even standard commands like "ls" produce output that make my toe nails
> roll up:
> 
> $ ls scripts/lib/bsp/substrate/target/arch/x86_64/recipes-kernel/linux/
> files
> {{ if kernel_choice == "linux-yocto_3.0": }} linux-yocto_3.0.bbappend
> {{ if kernel_choice == "linux-yocto_3.2": }} linux-yocto_3.2.bbappend
> {{ if kernel_choice == "linux-yocto-rt_3.0": }} linux-yocto-rt_3.0.bbappend
> {{ if kernel_choice == "linux-yocto-rt_3.2": }} linux-yocto-rt_3.2.bbappend
> kernel-list.noinstall
> $
> 
> PLEASE do not to this!
> 

If everyone really hates this, I'm perfectly happy with getting rid of
it - it's supposed to be a usability tool after all, and I definitely
don't want to be doing the opposite...

Tom

> Best regards,
> 
> Wolfgang Denk
> 

Re: Strange file names

[Detlev Zundel]

Hi Tom,

> It may seem strange, but filenames are just text after all, and I wanted
> to avoid having separate 'meta-files' describing that same logic that
> would also have to be kept in sync, so decided it would be less
> error-prone to keep that information in the filenames themselves.

It indeed is a possibility to encode "more than usual" into filenames,
but it certainly will have other effects also.  Now that the name of the
file itself contains "business logic" that is important for the system,
I wonder how this interacts with other parts of the (sometimes
unwritten) expectations of the development environment.  One thing which
immediately comes to my mind is the interaction with the version control
system.

Changing "an attribute" of such a file (which is coded in the inline
python code) will essentially look like the deletion and addition of a
file for version control.  Of course our version control (git) will be
clever enough to identify the file as being the same, but still we now
have an operation which was previously not used to change "business
logic", i.e. renaming files, doing just that.  So for tracing the
history of such an item I now need to trace the "rename history"
additionally to the "content history".

So in contrast to the original idea of "keeping the information in one
place", we now have two places, i.e. the content and the filename and
the latter is really a very uncommon place for that.

I do have a severe bad feeling in my stomach about going into this
(uncharted) territory and I do not know whether the potential problems
that we (yet) do not see are worth it.  So I would also like you to
reconsider that strategy.

Thanks
  Detlev

--
DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-40 Fax: (+49)-8142-66989-80 Email: dzu@denx.de

Re: Strange file names

[Tom Zanussi]

On Thu, 2012-03-29 at 10:54 +0200, Detlev Zundel wrote:
> Hi Tom,
> 
> > It may seem strange, but filenames are just text after all, and I wanted
> > to avoid having separate 'meta-files' describing that same logic that
> > would also have to be kept in sync, so decided it would be less
> > error-prone to keep that information in the filenames themselves.
> 
> It indeed is a possibility to encode "more than usual" into filenames,
> but it certainly will have other effects also.  Now that the name of the
> file itself contains "business logic" that is important for the system,
> I wonder how this interacts with other parts of the (sometimes
> unwritten) expectations of the development environment.  One thing which
> immediately comes to my mind is the interaction with the version control
> system.
> 
> Changing "an attribute" of such a file (which is coded in the inline
> python code) will essentially look like the deletion and addition of a
> file for version control.  Of course our version control (git) will be
> clever enough to identify the file as being the same, but still we now
> have an operation which was previously not used to change "business
> logic", i.e. renaming files, doing just that.  So for tracing the
> history of such an item I now need to trace the "rename history"
> additionally to the "content history".
> 
> So in contrast to the original idea of "keeping the information in one
> place", we now have two places, i.e. the content and the filename and
> the latter is really a very uncommon place for that.
> 
> I do have a severe bad feeling in my stomach about going into this
> (uncharted) territory and I do not know whether the potential problems
> that we (yet) do not see are worth it.  So I would also like you to
> reconsider that strategy.
> 

Yes, I think it would make sense for various reasons to get rid of it if
possible - it's not central to the mechanism, just convenient.

So the current uses of it are the following:

- having a particular filename match the machine name:
 
    ./powerpc/conf/machine/{{=machine}}.conf

- having a particular filename conditionally included or not:

    ./i386/recipes-kernel/linux/{{ if kernel_choice ==
"linux-yocto_3.2": }} linux-yocto_3.2.bbappend

For those two types of cases, I think it should be possible to move that
logic into the file itself using special filename and/or conditional
filename directives.

The other usage is to have directory names match the machine name for
instance:

 ./i386/recipes-graphics/xorg-xserver/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == "xserver_vesa": }} xorg.conf

In the case of directory names, since there's no file to put that
directive into, and we can't use the directory name any longer, we'll
have to again use some kind of special directory name directive and find
some file to put it into, probably just another .noinstall file, which
we already have a case of but that I wanted to get rid of.  There may be
a better way to do it, but we could keep those and do things that way in
the worst case.

Tom

> Thanks
>   Detlev
> 
> --
> DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-40 Fax: (+49)-8142-66989-80 Email: dzu@denx.de
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: Strange file names

[Tom Zanussi]

On Thu, 2012-03-29 at 18:02 +0200, Detlev Zundel wrote:
> Hi Tom,
> 
> thanks for listening to my concerns.  I have to admit though, that I
> wrote my mail because of my doubts about the usage of such unusal
> filenames, but I have yet to understand how you use the constructs.
> 
> > Yes, I think it would make sense for various reasons to get rid of it if
> > possible - it's not central to the mechanism, just convenient.
> >
> > So the current uses of it are the following:
> >
> > - having a particular filename match the machine name:
> >  
> >     ./powerpc/conf/machine/{{=machine}}.conf
> 
> Can you explain to me what this exactly does?  Does that file match
> every machine name?  If that is the case, then why do you need to match
> at all and not use a special file name?
> 

I guess 'match' is the wrong word - basically what happens when
processing the templates (and since the filenames themselves are part of
the 'templates', those too), the {{=machine}} in the name is replaced
with the name of the BSP.  So in the final set of files, supposing
you're creating a bsp named 'meta-foo', you end up with this file being
named:

./powerpc/conf/machine/foo.conf

which is what's expected in the BSP layout.

> > - having a particular filename conditionally included or not:
> >
> >     ./i386/recipes-kernel/linux/{{ if kernel_choice ==
> > "linux-yocto_3.2": }} linux-yocto_3.2.bbappend
> >
> > For those two types of cases, I think it should be possible to move that
> > logic into the file itself using special filename and/or conditional
> > filename directives.
> 
> I'm not sure if I understand fully, but something like (shell-syntax)
> "include ${kernel_choice}.bbappend" would work - modulo ignoring errors
> if the file is not found of course.
> 

So, similarly in this case, we have four files with similar names in
this dir, but only one of them should be used, and which one depends on
which kernel was selected.  The 'template processing' uses the code in
the filename to end up with that result.  So your suggestion is partly
fine in that we need some kind of 'include' to replace it, but it
doesn't contain the logic we need to choose only that file.  And it's
just replacing it with something that looks like executable code anyway.
But as I mentioned, I think this case also could me moved into the file
itself.

> > The other usage is to have directory names match the machine name for
> > instance:
> >
> >  ./i386/recipes-graphics/xorg-xserver/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == "xserver_vesa": }} xorg.conf
> 
> I fail to understand this for the same reason than the first item ;)
> 

This is actually a combination of the other two, replace the {{= with
'machine' but here it's a dirname, and conditionally select one of a set
of files.

Again, in this case, for the file in this case, we can move the
selection logic into the file, but for the dirname replacement we need
another place to contain it, which we can use the
aforementioned .noinstall files for, barring a better idea...

Tom

> Cheers
>   Detlev
> 
> --
> DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-40 Fax: (+49)-8142-66989-80 Email: dzu@denx.de

Re: Strange file names

[Detlev Zundel]

Hi Tom,

thanks for listening to my concerns.  I have to admit though, that I
wrote my mail because of my doubts about the usage of such unusal
filenames, but I have yet to understand how you use the constructs.

> Yes, I think it would make sense for various reasons to get rid of it if
> possible - it's not central to the mechanism, just convenient.
>
> So the current uses of it are the following:
>
> - having a particular filename match the machine name:
>  
>     ./powerpc/conf/machine/{{=machine}}.conf

Can you explain to me what this exactly does?  Does that file match
every machine name?  If that is the case, then why do you need to match
at all and not use a special file name?

> - having a particular filename conditionally included or not:
>
>     ./i386/recipes-kernel/linux/{{ if kernel_choice ==
> "linux-yocto_3.2": }} linux-yocto_3.2.bbappend
>
> For those two types of cases, I think it should be possible to move that
> logic into the file itself using special filename and/or conditional
> filename directives.

I'm not sure if I understand fully, but something like (shell-syntax)
"include ${kernel_choice}.bbappend" would work - modulo ignoring errors
if the file is not found of course.

> The other usage is to have directory names match the machine name for
> instance:
>
>  ./i386/recipes-graphics/xorg-xserver/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == "xserver_vesa": }} xorg.conf

I fail to understand this for the same reason than the first item ;)

Cheers
  Detlev

--
DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-40 Fax: (+49)-8142-66989-80 Email: dzu@denx.de