how to organize my patches for multiple kernels and multiple target boards?

how to organize my patches for multiple kernels and multiple target boards?

[Robert P. J. Day]

   (i posted a much lengthier version of this on oe-core recently, but i want
to cut it down and ask specific questions to clarify what i *think* is going
on.)

   i want to pull in an existing layer with recipes for linux-4.0.bb  
and linux-4.1.bb, and extend them with .bbappend files, to support two  
closely-related
machines i'm defining -- call them "mach1" and "mach2". AFAICT, my  
patches will
fall somewhere in a 3x3 matrix of possibilities:

   * 3 possibilities of applying against mach1, mach2 or both
   * 3 possibilities of applying against linux-4.0, linux-4.1 or both

so there's my 3x3 matrix.

   the obvious kernel recipe directory structure would be:

   linux/
     linux-4.0.bbappend
     linux-4.1.bbappend
     linux-4.0/            [4.0-specific patches]
     linux-4.1/            [4.1-specific patches]
     linux/                [patches that apply to both]

which suggests that both my .bbappend files would have to contain the line:

   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"

so the SRC_URI search path for linux-4.0.bbappend entries would be
prepended with:

   * linux-4.0/        [4.0-specific patches]
   * linux/            [patches that apply to both]

and similarly for linux.4.1.bbappend. how am i doing so far? this would
mean that, for each recipe, the more specific directory would be searched
before the general directory. but wait ... there's more.

   now i want to further categorize patches based on exclusive to mach1,
exclusive to mach2, or applicable to both, and since the machine name is
one of the entries in FILESOVERRIDES, i can extend the directory structure
as:

   linux-4.0/
     mach1/
     mach2/
   linux-4.1/
     mach1/
     mach2/
   linux/
     mach1/
     mach2/

and there's my 3x3 matrix of patches, correct? and here's where it gets
unclear.

   i really don't want to have to number all my patches with prefixes like
0001-, 0002- and so on, so what is the ordering of processing for .scc,
.cfg and .patch/.diff files? rather than just lump all the patches into
a single .scc file, i want to refine the patches across multiple .scc
files. is there an imposed order on SRC_URI entries, .scc files and so
on? that's probably all i need to finish this off.

rday

Re: how to organize my patches for multiple kernels and multiple target boards?

[Andrea Adami]

On Tue, Mar 1, 2016 at 12:19 AM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>   (i posted a much lengthier version of this on oe-core recently, but i want
> to cut it down and ask specific questions to clarify what i *think* is going
> on.)
>
>   i want to pull in an existing layer with recipes for linux-4.0.bb and
> linux-4.1.bb, and extend them with .bbappend files, to support two
> closely-related
> machines i'm defining -- call them "mach1" and "mach2". AFAICT, my patches
> will
> fall somewhere in a 3x3 matrix of possibilities:
>
>   * 3 possibilities of applying against mach1, mach2 or both
>   * 3 possibilities of applying against linux-4.0, linux-4.1 or both
>
> so there's my 3x3 matrix.
>
>   the obvious kernel recipe directory structure would be:
>
>   linux/
>     linux-4.0.bbappend
>     linux-4.1.bbappend
>     linux-4.0/            [4.0-specific patches]
>     linux-4.1/            [4.1-specific patches]
>     linux/                [patches that apply to both]
>
> which suggests that both my .bbappend files would have to contain the line:
>
>   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>
> so the SRC_URI search path for linux-4.0.bbappend entries would be
> prepended with:
>
>   * linux-4.0/        [4.0-specific patches]
>   * linux/            [patches that apply to both]
>
> and similarly for linux.4.1.bbappend. how am i doing so far? this would
> mean that, for each recipe, the more specific directory would be searched
> before the general directory. but wait ... there's more.
>
>   now i want to further categorize patches based on exclusive to mach1,
> exclusive to mach2, or applicable to both, and since the machine name is
> one of the entries in FILESOVERRIDES, i can extend the directory structure
> as:
>
>   linux-4.0/
>     mach1/
>     mach2/
>   linux-4.1/
>     mach1/
>     mach2/
>   linux/
>     mach1/
>     mach2/
>
> and there's my 3x3 matrix of patches, correct? and here's where it gets
> unclear.
>
>   i really don't want to have to number all my patches with prefixes like
> 0001-, 0002- and so on, so what is the ordering of processing for .scc,
> .cfg and .patch/.diff files? rather than just lump all the patches into
> a single .scc file, i want to refine the patches across multiple .scc
> files. is there an imposed order on SRC_URI entries, .scc files and so
> on? that's probably all i need to finish this off.
>
> rday

Robert,

in the past I have done pretty much the same: scc,cfg and patches all
packed in the recipe.
Please see these (outdated) layout examples for linux-yocto* that were
in meta-handheld.

for 3.10, using .cfg & .scc
http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dylan

Or simplified, for 3.14, using defconfig, with patches listed in SRC_URI
http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dizzy

Regards

Andrea

>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: how to organize my patches for multiple kernels and multiple target boards?

[Robert P. J. Day]

Quoting Andrea Adami <andrea.adami@gmail.com>:

> On Tue, Mar 1, 2016 at 12:19 AM, Robert P. J. Day  
> <rpjday@crashcourse.ca> wrote:
>>   (i posted a much lengthier version of this on oe-core recently, but i want
>> to cut it down and ask specific questions to clarify what i *think* is going
>> on.)
>>
>>   i want to pull in an existing layer with recipes for linux-4.0.bb and
>> linux-4.1.bb, and extend them with .bbappend files, to support two
>> closely-related
>> machines i'm defining -- call them "mach1" and "mach2". AFAICT, my patches
>> will
>> fall somewhere in a 3x3 matrix of possibilities:
>>
>>   * 3 possibilities of applying against mach1, mach2 or both
>>   * 3 possibilities of applying against linux-4.0, linux-4.1 or both
>>
>> so there's my 3x3 matrix.
>>
>>   the obvious kernel recipe directory structure would be:
>>
>>   linux/
>>     linux-4.0.bbappend
>>     linux-4.1.bbappend
>>     linux-4.0/            [4.0-specific patches]
>>     linux-4.1/            [4.1-specific patches]
>>     linux/                [patches that apply to both]
>>
>> which suggests that both my .bbappend files would have to contain the line:
>>
>>   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>>
>> so the SRC_URI search path for linux-4.0.bbappend entries would be
>> prepended with:
>>
>>   * linux-4.0/        [4.0-specific patches]
>>   * linux/            [patches that apply to both]
>>
>> and similarly for linux.4.1.bbappend. how am i doing so far? this would
>> mean that, for each recipe, the more specific directory would be searched
>> before the general directory. but wait ... there's more.
>>
>>   now i want to further categorize patches based on exclusive to mach1,
>> exclusive to mach2, or applicable to both, and since the machine name is
>> one of the entries in FILESOVERRIDES, i can extend the directory structure
>> as:
>>
>>   linux-4.0/
>>     mach1/
>>     mach2/
>>   linux-4.1/
>>     mach1/
>>     mach2/
>>   linux/
>>     mach1/
>>     mach2/
>>
>> and there's my 3x3 matrix of patches, correct? and here's where it gets
>> unclear.
>>
>>   i really don't want to have to number all my patches with prefixes like
>> 0001-, 0002- and so on, so what is the ordering of processing for .scc,
>> .cfg and .patch/.diff files? rather than just lump all the patches into
>> a single .scc file, i want to refine the patches across multiple .scc
>> files. is there an imposed order on SRC_URI entries, .scc files and so
>> on? that's probably all i need to finish this off.
>>
>> rday
>
> Robert,
>
> in the past I have done pretty much the same: scc,cfg and patches all
> packed in the recipe.
> Please see these (outdated) layout examples for linux-yocto* that were
> in meta-handheld.
>
> for 3.10, using .cfg & .scc
> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dylan
>
> Or simplified, for 3.14, using defconfig, with patches listed in SRC_URI
> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dizzy

   thanks, i'll check that out. first thing i want to be absolutely clear
on is, if i have multiple patch directories, i need to add them *all*
to the search path, as in:

   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"

and certainly in that order, as i want the more specific version
patches to be found first. so that bit is correct, yes?

   next, i'm still unclear on whether there is any enforced ordering
on the processing of .scc files. i know that some folks number their
patch files as 0001-*, 0002-* and so on, in order to enforce a
patching order (because the order that one specifies patch/diff files
in the SRC_URI doesn't mean anything, correct?)

   however, i would rather not use a numbering scheme like that because,
well, it's ugly, and given that i will have patches scattered all over
the patch directories and subdirectories on a per-kernel and a
per-target board basis, it just wouldn't make much sense.

   so my plan is to (predictably) group related patch files and .cfg
files into a number of .scc files but (again) is there any enforced
search order for .scc files? i'm assuming my setting for FILESEXTRAPATHS
and use of FILESOVERRIDES will kick in here ... will the order of
the .scc files in SRC_URI have any effect as well?

   also, once a .scc file is located, will the location of the listed
.cfg and .patch/.diff files inside it start a whole new search process
based on FILESEXTRAPATHS and FILESOVERRIDES?

   if this is all written down somewhere, just point me to it. thank
you kindly.

rday

p.s. all of this is in aid of trying to avoid ordering mishaps when
applying patches, but i'm guessing that if i design my .scc files
carefully to be logically self-contained, i can probably avoid
accidents like that in the first place.

Re: how to organize my patches for multiple kernels and multiple target boards?

[Bruce Ashfield]

On 16-03-01 05:44 AM, Robert P. J. Day wrote:
> Quoting Andrea Adami <andrea.adami@gmail.com>:
>
>> On Tue, Mar 1, 2016 at 12:19 AM, Robert P. J. Day
>> <rpjday@crashcourse.ca> wrote:
>>>   (i posted a much lengthier version of this on oe-core recently, but
>>> i want
>>> to cut it down and ask specific questions to clarify what i *think*
>>> is going
>>> on.)
>>>
>>>   i want to pull in an existing layer with recipes for linux-4.0.bb and
>>> linux-4.1.bb, and extend them with .bbappend files, to support two
>>> closely-related
>>> machines i'm defining -- call them "mach1" and "mach2". AFAICT, my
>>> patches
>>> will
>>> fall somewhere in a 3x3 matrix of possibilities:
>>>
>>>   * 3 possibilities of applying against mach1, mach2 or both
>>>   * 3 possibilities of applying against linux-4.0, linux-4.1 or both
>>>
>>> so there's my 3x3 matrix.
>>>
>>>   the obvious kernel recipe directory structure would be:
>>>
>>>   linux/
>>>     linux-4.0.bbappend
>>>     linux-4.1.bbappend
>>>     linux-4.0/            [4.0-specific patches]
>>>     linux-4.1/            [4.1-specific patches]
>>>     linux/                [patches that apply to both]
>>>
>>> which suggests that both my .bbappend files would have to contain the
>>> line:
>>>
>>>   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>>>
>>> so the SRC_URI search path for linux-4.0.bbappend entries would be
>>> prepended with:
>>>
>>>   * linux-4.0/        [4.0-specific patches]
>>>   * linux/            [patches that apply to both]
>>>
>>> and similarly for linux.4.1.bbappend. how am i doing so far? this would
>>> mean that, for each recipe, the more specific directory would be
>>> searched
>>> before the general directory. but wait ... there's more.
>>>
>>>   now i want to further categorize patches based on exclusive to mach1,
>>> exclusive to mach2, or applicable to both, and since the machine name is
>>> one of the entries in FILESOVERRIDES, i can extend the directory
>>> structure
>>> as:
>>>
>>>   linux-4.0/
>>>     mach1/
>>>     mach2/
>>>   linux-4.1/
>>>     mach1/
>>>     mach2/
>>>   linux/
>>>     mach1/
>>>     mach2/
>>>
>>> and there's my 3x3 matrix of patches, correct? and here's where it gets
>>> unclear.
>>>
>>>   i really don't want to have to number all my patches with prefixes
>>> like
>>> 0001-, 0002- and so on, so what is the ordering of processing for .scc,
>>> .cfg and .patch/.diff files? rather than just lump all the patches into
>>> a single .scc file, i want to refine the patches across multiple .scc
>>> files. is there an imposed order on SRC_URI entries, .scc files and so
>>> on? that's probably all i need to finish this off.
>>>
>>> rday
>>
>> Robert,
>>
>> in the past I have done pretty much the same: scc,cfg and patches all
>> packed in the recipe.
>> Please see these (outdated) layout examples for linux-yocto* that were
>> in meta-handheld.
>>
>> for 3.10, using .cfg & .scc
>> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dylan
>>
>>
>> Or simplified, for 3.14, using defconfig, with patches listed in SRC_URI
>> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dizzy
>>
>
>    thanks, i'll check that out. first thing i want to be absolutely clear
> on is, if i have multiple patch directories, i need to add them *all*
> to the search path, as in:
>
>    FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>
> and certainly in that order, as i want the more specific version
> patches to be found first. so that bit is correct, yes?
>
>    next, i'm still unclear on whether there is any enforced ordering
> on the processing of .scc files. i know that some folks number their
> patch files as 0001-*, 0002-* and so on, in order to enforce a
> patching order (because the order that one specifies patch/diff files
> in the SRC_URI doesn't mean anything, correct?)

I'm waiting for someone to slap me for saying something so definitive,
but here it goes anyay ... The order in the SRC_URI definitely matters.
The order that they are specified is the order the patches are applied.
Otherwise, a patch stack of any depth wouldn't be possible. Only if you
are doing wildcard patches and rely on shell globbing would numbering
be critical.

>
>    however, i would rather not use a numbering scheme like that because,
> well, it's ugly, and given that i will have patches scattered all over
> the patch directories and subdirectories on a per-kernel and a
> per-target board basis, it just wouldn't make much sense.

Agreed.

>
>    so my plan is to (predictably) group related patch files and .cfg
> files into a number of .scc files but (again) is there any enforced
> search order for .scc files? i'm assuming my setting for FILESEXTRAPATHS
> and use of FILESOVERRIDES will kick in here ... will the order of
> the .scc files in SRC_URI have any effect as well?

.scc files will be processed in the order they are found on the
SRC_URI, and then the patches the order they are within the .scc
files.

>
>    also, once a .scc file is located, will the location of the listed
> .cfg and .patch/.diff files inside it start a whole new search process
> based on FILESEXTRAPATHS and FILESOVERRIDES?

The search path is created relative to the .scc file that is referencing
a patch, and across any directory that has a .scc file in the SRC_URI.
The .scc feature descriptions are self contained, hence "reaching outside"
to a parent, or other directory structure isn't a good thing. Just list
things on the SRC_URI if that is required.

>
>    if this is all written down somewhere, just point me to it. thank
> you kindly.

IIRC it is in the kernel development manual (the .scc file parts), but
the SRC_URI and patch order is common to any recipe that bitbake/oe-core
process.

>
> rday
>
> p.s. all of this is in aid of trying to avoid ordering mishaps when
> applying patches, but i'm guessing that if i design my .scc files
> carefully to be logically self-contained, i can probably avoid
> accidents like that in the first place.

Correct. The .scc files, kernel-cache and management found within
the files was created to manage hundreds of patches for overlapping
boards, much like you are describing.

In the end, when complexity gets really high, a git repo with branches
is probably easier ... and with that description, I've just covered
how patch management with .scc files leads to the linux-yocto tree :)

Bruce

>
>

Re: how to organize my patches for multiple kernels and multiple target boards?

[Robert P. J. Day]

Quoting Bruce Ashfield <bruce.ashfield@windriver.com>:

> On 16-03-01 05:44 AM, Robert P. J. Day wrote:
>> Quoting Andrea Adami <andrea.adami@gmail.com>:
>>
>>> On Tue, Mar 1, 2016 at 12:19 AM, Robert P. J. Day
>>> <rpjday@crashcourse.ca> wrote:
>>>>  (i posted a much lengthier version of this on oe-core recently, but
>>>> i want
>>>> to cut it down and ask specific questions to clarify what i *think*
>>>> is going
>>>> on.)
>>>>
>>>>  i want to pull in an existing layer with recipes for linux-4.0.bb and
>>>> linux-4.1.bb, and extend them with .bbappend files, to support two
>>>> closely-related
>>>> machines i'm defining -- call them "mach1" and "mach2". AFAICT, my
>>>> patches
>>>> will
>>>> fall somewhere in a 3x3 matrix of possibilities:
>>>>
>>>>  * 3 possibilities of applying against mach1, mach2 or both
>>>>  * 3 possibilities of applying against linux-4.0, linux-4.1 or both
>>>>
>>>> so there's my 3x3 matrix.
>>>>
>>>>  the obvious kernel recipe directory structure would be:
>>>>
>>>>  linux/
>>>>    linux-4.0.bbappend
>>>>    linux-4.1.bbappend
>>>>    linux-4.0/            [4.0-specific patches]
>>>>    linux-4.1/            [4.1-specific patches]
>>>>    linux/                [patches that apply to both]
>>>>
>>>> which suggests that both my .bbappend files would have to contain the
>>>> line:
>>>>
>>>>  FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>>>>
>>>> so the SRC_URI search path for linux-4.0.bbappend entries would be
>>>> prepended with:
>>>>
>>>>  * linux-4.0/        [4.0-specific patches]
>>>>  * linux/            [patches that apply to both]
>>>>
>>>> and similarly for linux.4.1.bbappend. how am i doing so far? this would
>>>> mean that, for each recipe, the more specific directory would be
>>>> searched
>>>> before the general directory. but wait ... there's more.
>>>>
>>>>  now i want to further categorize patches based on exclusive to mach1,
>>>> exclusive to mach2, or applicable to both, and since the machine name is
>>>> one of the entries in FILESOVERRIDES, i can extend the directory
>>>> structure
>>>> as:
>>>>
>>>>  linux-4.0/
>>>>    mach1/
>>>>    mach2/
>>>>  linux-4.1/
>>>>    mach1/
>>>>    mach2/
>>>>  linux/
>>>>    mach1/
>>>>    mach2/
>>>>
>>>> and there's my 3x3 matrix of patches, correct? and here's where it gets
>>>> unclear.
>>>>
>>>>  i really don't want to have to number all my patches with prefixes
>>>> like
>>>> 0001-, 0002- and so on, so what is the ordering of processing for .scc,
>>>> .cfg and .patch/.diff files? rather than just lump all the patches into
>>>> a single .scc file, i want to refine the patches across multiple .scc
>>>> files. is there an imposed order on SRC_URI entries, .scc files and so
>>>> on? that's probably all i need to finish this off.
>>>>
>>>> rday
>>>
>>> Robert,
>>>
>>> in the past I have done pretty much the same: scc,cfg and patches all
>>> packed in the recipe.
>>> Please see these (outdated) layout examples for linux-yocto* that were
>>> in meta-handheld.
>>>
>>> for 3.10, using .cfg & .scc
>>> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dylan
>>>
>>>
>>> Or simplified, for 3.14, using defconfig, with patches listed in SRC_URI
>>> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dizzy
>>>
>>
>>   thanks, i'll check that out. first thing i want to be absolutely clear
>> on is, if i have multiple patch directories, i need to add them *all*
>> to the search path, as in:
>>
>>   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>>
>> and certainly in that order, as i want the more specific version
>> patches to be found first. so that bit is correct, yes?
>>
>>   next, i'm still unclear on whether there is any enforced ordering
>> on the processing of .scc files. i know that some folks number their
>> patch files as 0001-*, 0002-* and so on, in order to enforce a
>> patching order (because the order that one specifies patch/diff files
>> in the SRC_URI doesn't mean anything, correct?)
>
> I'm waiting for someone to slap me for saying something so definitive,
> but here it goes anyay ... The order in the SRC_URI definitely matters.
> The order that they are specified is the order the patches are applied.
> Otherwise, a patch stack of any depth wouldn't be possible. Only if you
> are doing wildcard patches and rely on shell globbing would numbering
> be critical.

eggcellent ... i assume that SRC_URI ordering is imposed on *all* SRC_URI
items, that being .scc, .cfg and .patch/.diff files, as in SRC_URI is
processed strictly in order for all of its constituent items? i always
assumed as much, i just don't recall ever seeing that written down. and,
uh, i suspect scott rifenbark can testify that i'm moderately familiar
with the manuals. :-)

>>   however, i would rather not use a numbering scheme like that because,
>> well, it's ugly, and given that i will have patches scattered all over
>> the patch directories and subdirectories on a per-kernel and a
>> per-target board basis, it just wouldn't make much sense.
>
> Agreed.
>
>>   so my plan is to (predictably) group related patch files and .cfg
>> files into a number of .scc files but (again) is there any enforced
>> search order for .scc files? i'm assuming my setting for FILESEXTRAPATHS
>> and use of FILESOVERRIDES will kick in here ... will the order of
>> the .scc files in SRC_URI have any effect as well?
>
> .scc files will be processed in the order they are found on the
> SRC_URI, and then the patches the order they are within the .scc
> files.

getting better and better ...

>>   also, once a .scc file is located, will the location of the listed
>> .cfg and .patch/.diff files inside it start a whole new search process
>> based on FILESEXTRAPATHS and FILESOVERRIDES?
>
> The search path is created relative to the .scc file that is referencing
> a patch, and across any directory that has a .scc file in the SRC_URI.
> The .scc feature descriptions are self contained, hence "reaching outside"
> to a parent, or other directory structure isn't a good thing. Just list
> things on the SRC_URI if that is required.

   ah, so to clarify, if i refer to a .scc file, my FILESEXTRAPATHS and
FILESOVERRIDES will kick in to *find* it, but once found, its internal
references will be treated as local to where the .scc file was found?
good, that makes sense, i like that.

>>   if this is all written down somewhere, just point me to it. thank
>> you kindly.
>
> IIRC it is in the kernel development manual (the .scc file parts), but
> the SRC_URI and patch order is common to any recipe that bitbake/oe-core
> process.

>> p.s. all of this is in aid of trying to avoid ordering mishaps when
>> applying patches, but i'm guessing that if i design my .scc files
>> carefully to be logically self-contained, i can probably avoid
>> accidents like that in the first place.
>
> Correct. The .scc files, kernel-cache and management found within
> the files was created to manage hundreds of patches for overlapping
> boards, much like you are describing.
>
> In the end, when complexity gets really high, a git repo with branches
> is probably easier ... and with that description, I've just covered
> how patch management with .scc files leads to the linux-yocto tree :)
>
> Bruce

   smugness becomes you. :-)

rday

Re: how to organize my patches for multiple kernels and multiple target boards?

[Bruce Ashfield]

On 2016-03-01 3:19 PM, Robert P. J. Day wrote:
> Quoting Bruce Ashfield <bruce.ashfield@windriver.com>:
>
>> On 16-03-01 05:44 AM, Robert P. J. Day wrote:
>>> Quoting Andrea Adami <andrea.adami@gmail.com>:
>>>
>>>> On Tue, Mar 1, 2016 at 12:19 AM, Robert P. J. Day
>>>> <rpjday@crashcourse.ca> wrote:
>>>>>  (i posted a much lengthier version of this on oe-core recently, but
>>>>> i want
>>>>> to cut it down and ask specific questions to clarify what i *think*
>>>>> is going
>>>>> on.)
>>>>>
>>>>>  i want to pull in an existing layer with recipes for linux-4.0.bb and
>>>>> linux-4.1.bb, and extend them with .bbappend files, to support two
>>>>> closely-related
>>>>> machines i'm defining -- call them "mach1" and "mach2". AFAICT, my
>>>>> patches
>>>>> will
>>>>> fall somewhere in a 3x3 matrix of possibilities:
>>>>>
>>>>>  * 3 possibilities of applying against mach1, mach2 or both
>>>>>  * 3 possibilities of applying against linux-4.0, linux-4.1 or both
>>>>>
>>>>> so there's my 3x3 matrix.
>>>>>
>>>>>  the obvious kernel recipe directory structure would be:
>>>>>
>>>>>  linux/
>>>>>    linux-4.0.bbappend
>>>>>    linux-4.1.bbappend
>>>>>    linux-4.0/            [4.0-specific patches]
>>>>>    linux-4.1/            [4.1-specific patches]
>>>>>    linux/                [patches that apply to both]
>>>>>
>>>>> which suggests that both my .bbappend files would have to contain the
>>>>> line:
>>>>>
>>>>>  FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>>>>>
>>>>> so the SRC_URI search path for linux-4.0.bbappend entries would be
>>>>> prepended with:
>>>>>
>>>>>  * linux-4.0/        [4.0-specific patches]
>>>>>  * linux/            [patches that apply to both]
>>>>>
>>>>> and similarly for linux.4.1.bbappend. how am i doing so far? this
>>>>> would
>>>>> mean that, for each recipe, the more specific directory would be
>>>>> searched
>>>>> before the general directory. but wait ... there's more.
>>>>>
>>>>>  now i want to further categorize patches based on exclusive to mach1,
>>>>> exclusive to mach2, or applicable to both, and since the machine
>>>>> name is
>>>>> one of the entries in FILESOVERRIDES, i can extend the directory
>>>>> structure
>>>>> as:
>>>>>
>>>>>  linux-4.0/
>>>>>    mach1/
>>>>>    mach2/
>>>>>  linux-4.1/
>>>>>    mach1/
>>>>>    mach2/
>>>>>  linux/
>>>>>    mach1/
>>>>>    mach2/
>>>>>
>>>>> and there's my 3x3 matrix of patches, correct? and here's where it
>>>>> gets
>>>>> unclear.
>>>>>
>>>>>  i really don't want to have to number all my patches with prefixes
>>>>> like
>>>>> 0001-, 0002- and so on, so what is the ordering of processing for
>>>>> .scc,
>>>>> .cfg and .patch/.diff files? rather than just lump all the patches
>>>>> into
>>>>> a single .scc file, i want to refine the patches across multiple .scc
>>>>> files. is there an imposed order on SRC_URI entries, .scc files and so
>>>>> on? that's probably all i need to finish this off.
>>>>>
>>>>> rday
>>>>
>>>> Robert,
>>>>
>>>> in the past I have done pretty much the same: scc,cfg and patches all
>>>> packed in the recipe.
>>>> Please see these (outdated) layout examples for linux-yocto* that were
>>>> in meta-handheld.
>>>>
>>>> for 3.10, using .cfg & .scc
>>>> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dylan
>>>>
>>>>
>>>>
>>>> Or simplified, for 3.14, using defconfig, with patches listed in
>>>> SRC_URI
>>>> http://cgit.openembedded.org/meta-handheld/tree/recipes-kernel/linux?h=dizzy
>>>>
>>>>
>>>
>>>   thanks, i'll check that out. first thing i want to be absolutely clear
>>> on is, if i have multiple patch directories, i need to add them *all*
>>> to the search path, as in:
>>>
>>>   FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>>>
>>> and certainly in that order, as i want the more specific version
>>> patches to be found first. so that bit is correct, yes?
>>>
>>>   next, i'm still unclear on whether there is any enforced ordering
>>> on the processing of .scc files. i know that some folks number their
>>> patch files as 0001-*, 0002-* and so on, in order to enforce a
>>> patching order (because the order that one specifies patch/diff files
>>> in the SRC_URI doesn't mean anything, correct?)
>>
>> I'm waiting for someone to slap me for saying something so definitive,
>> but here it goes anyay ... The order in the SRC_URI definitely matters.
>> The order that they are specified is the order the patches are applied.
>> Otherwise, a patch stack of any depth wouldn't be possible. Only if you
>> are doing wildcard patches and rely on shell globbing would numbering
>> be critical.
>
> eggcellent ... i assume that SRC_URI ordering is imposed on *all* SRC_URI
> items, that being .scc, .cfg and .patch/.diff files, as in SRC_URI is
> processed strictly in order for all of its constituent items? i always

It is. If it isn't, that's a bug.

> assumed as much, i just don't recall ever seeing that written down. and,
> uh, i suspect scott rifenbark can testify that i'm moderately familiar
> with the manuals. :-)
>
>>>   however, i would rather not use a numbering scheme like that because,
>>> well, it's ugly, and given that i will have patches scattered all over
>>> the patch directories and subdirectories on a per-kernel and a
>>> per-target board basis, it just wouldn't make much sense.
>>
>> Agreed.
>>
>>>   so my plan is to (predictably) group related patch files and .cfg
>>> files into a number of .scc files but (again) is there any enforced
>>> search order for .scc files? i'm assuming my setting for FILESEXTRAPATHS
>>> and use of FILESOVERRIDES will kick in here ... will the order of
>>> the .scc files in SRC_URI have any effect as well?
>>
>> .scc files will be processed in the order they are found on the
>> SRC_URI, and then the patches the order they are within the .scc
>> files.
>
> getting better and better ...
>
>>>   also, once a .scc file is located, will the location of the listed
>>> .cfg and .patch/.diff files inside it start a whole new search process
>>> based on FILESEXTRAPATHS and FILESOVERRIDES?
>>
>> The search path is created relative to the .scc file that is referencing
>> a patch, and across any directory that has a .scc file in the SRC_URI.
>> The .scc feature descriptions are self contained, hence "reaching
>> outside"
>> to a parent, or other directory structure isn't a good thing. Just list
>> things on the SRC_URI if that is required.
>
>    ah, so to clarify, if i refer to a .scc file, my FILESEXTRAPATHS and
> FILESOVERRIDES will kick in to *find* it, but once found, its internal
> references will be treated as local to where the .scc file was found?
> good, that makes sense, i like that.

Correct.

>
>>>   if this is all written down somewhere, just point me to it. thank
>>> you kindly.
>>
>> IIRC it is in the kernel development manual (the .scc file parts), but
>> the SRC_URI and patch order is common to any recipe that bitbake/oe-core
>> process.
>
>>> p.s. all of this is in aid of trying to avoid ordering mishaps when
>>> applying patches, but i'm guessing that if i design my .scc files
>>> carefully to be logically self-contained, i can probably avoid
>>> accidents like that in the first place.
>>
>> Correct. The .scc files, kernel-cache and management found within
>> the files was created to manage hundreds of patches for overlapping
>> boards, much like you are describing.
>>
>> In the end, when complexity gets really high, a git repo with branches
>> is probably easier ... and with that description, I've just covered
>> how patch management with .scc files leads to the linux-yocto tree :)
>>
>> Bruce
>
>    smugness becomes you. :-)

Ha! :)

Bruce

>
> rday
>
>

Re: how to organize my patches for multiple kernels and multiple target boards?

[Robert P. J. Day]

Quoting Bruce Ashfield <bruce.ashfield@windriver.com>:

> On 16-03-01 05:44 AM, Robert P. J. Day wrote:

>>   also, once a .scc file is located, will the location of the listed
>> .cfg and .patch/.diff files inside it start a whole new search process
>> based on FILESEXTRAPATHS and FILESOVERRIDES?

> The search path is created relative to the .scc file that is referencing
> a patch, and across any directory that has a .scc file in the SRC_URI.
> The .scc feature descriptions are self contained, hence "reaching outside"
> to a parent, or other directory structure isn't a good thing. Just list
> things on the SRC_URI if that is required.

   one more question, if i might, as i'm still unclear on what flexibility
i have with finding .cfg or .patch files from a .scc file.

   imagine i have a number of kernel .bbappend files and corresponding
patch directories:

   * linux-4.0.bbappend and linux-4.0/
   * linux-4.1.bbappend and linux-4.1/
   * linux-4.2.bbappend and linux-4.2/

and so on, as well as wanting to refer to five machines, "m1" through
"m5". consider the following possibility related specifically to
kernel version 4.1:

   linux-4.1/
     rday.scc
     1.patch
     2.patch
     3.patch
     4.patch
     m3/
       4.patch

so imagine my linux-4.1.bbappend file does SRC_URI += "rday.scc" --
when i do a build for kernel 4.1, the search process should locate
the rday.scc file in linux-4.1/ (as long as i have no other
higher-priority FILESOVERRIDES that get in the way, of course).

   now if rday.scc contains references to all four patches and i'm
building for, say, machine "m1", it makes sense that all four
of those patches directly under linux-4.1/ will be the ones
included, correct?

   but if, instead, i was building for machine "m3", as you can
see, it would be nice if the FILESOVERRIDES feature would kick in
and select the machine-specific patch "m3/4.patch. so all of the
regular patches would be used, *unless* i was building for m3,
at which point the patch m3/4.patch would override the "generic"
4.patch. (the same logic would apply to .cfg files, too, of course.)

   is that what would happen? better yet, is that anything i
should even be contemplating? it's not as if i need that feature
right this instant, but it would be nice to know it's available
just in case.

   even weirder, could i get away with something like this?

   linux-4.1/
     rday.scc
     1.patch
     2.patch
     3.patch
     4.patch
     m3/
       rday.scc
       4.patch

once again, my .bbappend file would "SRC_URI += rday.scc", and if
i'm building for kernel 4.1, it should find the one directly under
linux-4.1/, *unless* my target machine is "m3", at which point
the file m3/rday.scc would take precedence, which would pick up
the specific patch file m3/4.patch, but would use the higher-level
generic patch files for all others.

   the first case i think has value and i'd like to know how to do
it; the second case is admittedly weirder and i'm not sure i
want to defend even *trying* to do it, but i figured i'd ask.

rday

Re: how to organize my patches for multiple kernels and multiple target boards?

[Bruce Ashfield]

On 2016-03-02 10:41 AM, Robert P. J. Day wrote:
> Quoting Bruce Ashfield <bruce.ashfield@windriver.com>:
>
>> On 16-03-01 05:44 AM, Robert P. J. Day wrote:
>
>>>   also, once a .scc file is located, will the location of the listed
>>> .cfg and .patch/.diff files inside it start a whole new search process
>>> based on FILESEXTRAPATHS and FILESOVERRIDES?
>
>> The search path is created relative to the .scc file that is referencing
>> a patch, and across any directory that has a .scc file in the SRC_URI.
>> The .scc feature descriptions are self contained, hence "reaching
>> outside"
>> to a parent, or other directory structure isn't a good thing. Just list
>> things on the SRC_URI if that is required.
>
>    one more question, if i might, as i'm still unclear on what flexibility
> i have with finding .cfg or .patch files from a .scc file.
>
>    imagine i have a number of kernel .bbappend files and corresponding
> patch directories:
>
>    * linux-4.0.bbappend and linux-4.0/
>    * linux-4.1.bbappend and linux-4.1/
>    * linux-4.2.bbappend and linux-4.2/
>
> and so on, as well as wanting to refer to five machines, "m1" through
> "m5". consider the following possibility related specifically to
> kernel version 4.1:
>
>    linux-4.1/
>      rday.scc
>      1.patch
>      2.patch
>      3.patch
>      4.patch
>      m3/
>        4.patch
>
> so imagine my linux-4.1.bbappend file does SRC_URI += "rday.scc" --
> when i do a build for kernel 4.1, the search process should locate
> the rday.scc file in linux-4.1/ (as long as i have no other
> higher-priority FILESOVERRIDES that get in the way, of course).
>
>    now if rday.scc contains references to all four patches and i'm
> building for, say, machine "m1", it makes sense that all four
> of those patches directly under linux-4.1/ will be the ones
> included, correct?

Correct. Assuming rday.scc and the patch files are all within that
directory, it would be:

patch 1.patch
patch 2.patch
... etc.

And that would find the ones local to the .scc file.

>
>    but if, instead, i was building for machine "m3", as you can
> see, it would be nice if the FILESOVERRIDES feature would kick in
> and select the machine-specific patch "m3/4.patch. so all of the
> regular patches would be used, *unless* i was building for m3,
> at which point the patch m3/4.patch would override the "generic"
> 4.patch. (the same logic would apply to .cfg files, too, of course.)
>
>    is that what would happen? better yet, is that anything i
> should even be contemplating? it's not as if i need that feature
> right this instant, but it would be nice to know it's available
> just in case.

That doesn't happen. Since the searching for patches is not tied
to the fetcher searching and ordering directly.

You can switch on matchines within a .scc file, but that's not
really all that common.

>
>    even weirder, could i get away with something like this?
>
>    linux-4.1/
>      rday.scc
>      1.patch
>      2.patch
>      3.patch
>      4.patch
>      m3/
>        rday.scc
>        4.patch
>
> once again, my .bbappend file would "SRC_URI += rday.scc", and if
> i'm building for kernel 4.1, it should find the one directly under
> linux-4.1/, *unless* my target machine is "m3", at which point
> the file m3/rday.scc would take precedence, which would pick up
> the specific patch file m3/4.patch, but would use the higher-level
> generic patch files for all others.


Assuming the fetch found m3/rday.scc in the search paths first, it
would be the one selected.

But then the patch references would be relative to m3/, so you wouldn't
even find "patch 1.patch", etc.


The way this is typically configured if you aren't using a kernel-cache
structure is:

  linux-<foo>/
      machine-type1.scc
      machine-type2.scc
      machine-type3.scc
      common/
        1.patch
        2.patch
        3.patch
        4.patch
       type3/
        4.patch

And then you select the one that matches in your SRC_URI, or if the
.scc files have: "define KMACHINE type1", and you put them ALL on
the SRC_URI, the system will actually pick only the one that
matches the set $MACHINE.

 From there, that matching .scc file does all it's own includes of
patches and configuration blocks, etc.

Bruce

>
>    the first case i think has value and i'd like to know how to do
> it; the second case is admittedly weirder and i'm not sure i
> want to defend even *trying* to do it, but i figured i'd ask.
>
> rday
>
>

Re: how to organize my patches for multiple kernels and multiple target boards?

[Bruce Ashfield]

On 2016-02-29 6:19 PM, Robert P. J. Day wrote:
>    (i posted a much lengthier version of this on oe-core recently, but i
> want
> to cut it down and ask specific questions to clarify what i *think* is
> going
> on.)
>
>    i want to pull in an existing layer with recipes for linux-4.0.bb and
> linux-4.1.bb, and extend them with .bbappend files, to support two
> closely-related
> machines i'm defining -- call them "mach1" and "mach2". AFAICT, my
> patches will
> fall somewhere in a 3x3 matrix of possibilities:
>
>    * 3 possibilities of applying against mach1, mach2 or both
>    * 3 possibilities of applying against linux-4.0, linux-4.1 or both
>
> so there's my 3x3 matrix.
>
>    the obvious kernel recipe directory structure would be:
>
>    linux/
>      linux-4.0.bbappend
>      linux-4.1.bbappend
>      linux-4.0/            [4.0-specific patches]
>      linux-4.1/            [4.1-specific patches]
>      linux/                [patches that apply to both]
>
> which suggests that both my .bbappend files would have to contain the line:
>
>    FILESEXTRAPATHS_prepend := "${THISDIR}/${BP}:${THISDIR}/${BPN}"
>
> so the SRC_URI search path for linux-4.0.bbappend entries would be
> prepended with:
>
>    * linux-4.0/        [4.0-specific patches]
>    * linux/            [patches that apply to both]
>
> and similarly for linux.4.1.bbappend. how am i doing so far? this would
> mean that, for each recipe, the more specific directory would be searched
> before the general directory. but wait ... there's more.
>
>    now i want to further categorize patches based on exclusive to mach1,
> exclusive to mach2, or applicable to both, and since the machine name is
> one of the entries in FILESOVERRIDES, i can extend the directory structure
> as:
>
>    linux-4.0/
>      mach1/
>      mach2/
>    linux-4.1/
>      mach1/
>      mach2/
>    linux/
>      mach1/
>      mach2/
>
> and there's my 3x3 matrix of patches, correct? and here's where it gets
> unclear.
>
>    i really don't want to have to number all my patches with prefixes like
> 0001-, 0002- and so on, so what is the ordering of processing for .scc,
> .cfg and .patch/.diff files? rather than just lump all the patches into
> a single .scc file, i want to refine the patches across multiple .scc
> files. is there an imposed order on SRC_URI entries, .scc files and so
> on? that's probably all i need to finish this off.

No matter what you method you choose, ordering is as they appear in the
SRC_URI (and normal variable evaluation rules apply)/.

Having maintained more than a few kernel's, my warning is that depending
on your patches, the advantages of mixing version independent patches
with version specific (and board ones) can end up causing a lot of extra
work and maintenance pain. It of course all comes down to what parts of
the kernel they touch, but assuming a normal set of patches you'll find
that you end up tweaking things for order, and then moving patches into
version/board specific places, etc. In particular if you update the base
kernel's to have things like -stable, or other patches that touch lots
of code.

Outside of the kernel version handling (see my warning above), managing
the patches by the SRC_URI works, as would .scc files. Since the entire
point of .scc files is to define a board entry point (the top level .scc
file), and then have it include common patches, configs, etc, all from
that file.

There is another alternative to that management of patches/configs/boards,
and you can create a kernel-cache directory and refer to it on the
SRC_URI. In master, that's how all the boards and patches are managed.

Cheers,

Bruce



>
> rday
>
>
>