Building "restricted" source code

Building "restricted" source code

[Jerrod Peach]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

As part of my company's firmware builds, we have to build some code that
only a handful of developers are allowed to view.  We call this restricted
source code.  Getting our official "system" builds to build this code isn't
a problem.  What is a problem is a regular developer's build of this code.

Imagine this scenario: The restricted source depends upon eglibc.  Our
low-level team, who doesn't have access to the restricted source, updates
the recipe for eglibc.  The hash for the restricted package is permuted,
and so they can't get an sstate hit and are required to rebuild the source.
 But, since they can't check out the source, they can't build it.  This
would cause a build failure.

Is anyone dealing with this scenario while using Yocto currently?  If so,
how?  I know it may be unlikely that many people are hitting this since
Yocto is primarily used to build open-source code, but I thought I'd take a
shot in the dark and hope for the best.  :-)

[-- Attachment #2: Type: text/html, Size: 1085 bytes --]

Re: Building "restricted" source code

[Mark Hatle]

On 3/26/13 7:53 AM, Jerrod Peach wrote:
> As part of my company's firmware builds, we have to build some code that only a
> handful of developers are allowed to view.  We call this restricted source code.
>   Getting our official "system" builds to build this code isn't a problem.  What
> is a problem is a regular developer's build of this code.
>
> Imagine this scenario: The restricted source depends upon eglibc.  Our low-level
> team, who doesn't have access to the restricted source, updates the recipe for
> eglibc.  The hash for the restricted package is permuted, and so they can't get
> an sstate hit and are required to rebuild the source.  But, since they can't
> check out the source, they can't build it.  This would cause a build failure.
>
> Is anyone dealing with this scenario while using Yocto currently?  If so, how?
>   I know it may be unlikely that many people are hitting this since Yocto is
> primarily used to build open-source code, but I thought I'd take a shot in the
> dark and hope for the best.  :-)
>

Two ways I know of doing this.  Slightly different way of doing things:

1) If the code does -not- rely on outside influences, such as eglibc.  Configure 
the recipe and pretty much ignore everything else with a vardepsexp.  Then ship 
the sstate-cache files that cover the compiled code.  (Along with the original 
recipe...)

2) For code that DOES rely on outside influences.. create a custom recipe that 
either downloads the source and builds it [if the user has access to the 
source], or will pull the binaries from a specific location and simply 
install/package them.   This is actually the more common approach.

(To seed that location, you can extract the items from your restricted build -- 
or build it outside the tree using an SDK or similar.)

--Mark

> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Building "restricted" source code

[Jerrod Peach]

[-- Attachment #1: Type: text/plain, Size: 4001 bytes --]

Mark,

Option 1 isn't a very big concern since it's likely only developers working
on the restricted code will cause its hashes to permute.

Option 2 is precisely what we were thinking about doing.  The concern,
though, is what happens when a developer who doesn't have access to the
code changes a dependency of the restricted code in such a way that it
actually affects the restricted code.  That developer is unable to test the
effects of his changes without pushing them to a back-end build, which
would involve pushing untested changes somewhere.  I don't love that
thought.

Another option would be to make the recipe smart enough to build restricted
source on a different machine from the user's using an SDK (which is what I
think you were suggesting), but that involves transferring the user's
uncommitted code and/or built objects to the separate machine, adding a
fairly significant layer of complexity (and likely a significant amount of
time) to what is otherwise a relatively straightforward build process.

I'm not the biggest fan of any of these options presently, but if we have
to choose one of these routes, we will.  (We were likely going to use what
you presented as your second option, but that's still going to involve
someone updating recipes for restricted source every time new binaries
become available.)  I do very much appreciate the input and I'm glad to
know the ideas we came up with aren't completely crazy, since someone else
has thought of them too ;-)  Any other thoughts on this issue?

Kind regards,

Jerrod


On Tue, Mar 26, 2013 at 3:21 PM, Mark Hatle <mark.hatle@windriver.com>wrote:

> On 3/26/13 7:53 AM, Jerrod Peach wrote:
>
>> As part of my company's firmware builds, we have to build some code that
>> only a
>> handful of developers are allowed to view.  We call this restricted
>> source code.
>>   Getting our official "system" builds to build this code isn't a
>> problem.  What
>> is a problem is a regular developer's build of this code.
>>
>> Imagine this scenario: The restricted source depends upon eglibc.  Our
>> low-level
>> team, who doesn't have access to the restricted source, updates the
>> recipe for
>> eglibc.  The hash for the restricted package is permuted, and so they
>> can't get
>> an sstate hit and are required to rebuild the source.  But, since they
>> can't
>> check out the source, they can't build it.  This would cause a build
>> failure.
>>
>> Is anyone dealing with this scenario while using Yocto currently?  If so,
>> how?
>>   I know it may be unlikely that many people are hitting this since Yocto
>> is
>> primarily used to build open-source code, but I thought I'd take a shot
>> in the
>> dark and hope for the best.  :-)
>>
>>
> Two ways I know of doing this.  Slightly different way of doing things:
>
> 1) If the code does -not- rely on outside influences, such as eglibc.
>  Configure the recipe and pretty much ignore everything else with a
> vardepsexp.  Then ship the sstate-cache files that cover the compiled code.
>  (Along with the original recipe...)
>
> 2) For code that DOES rely on outside influences.. create a custom recipe
> that either downloads the source and builds it [if the user has access to
> the source], or will pull the binaries from a specific location and simply
> install/package them.   This is actually the more common approach.
>
> (To seed that location, you can extract the items from your restricted
> build -- or build it outside the tree using an SDK or similar.)
>
> --Mark
>
>  ______________________________**_________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.**org/listinfo/yocto<https://lists.yoctoproject.org/listinfo/yocto>
>>
>>
> ______________________________**_________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.**org/listinfo/yocto<https://lists.yoctoproject.org/listinfo/yocto>
>

[-- Attachment #2: Type: text/html, Size: 5026 bytes --]

Re: Building "restricted" source code

[Samuel Stirtzel]

While this may not resolve your problem, these are my 0.2 cents.

I assume you already know this:
The hash does not need to be permuted between dependency updates, as
(e.g.) eglibc "promises" ABI and API stability between major releases
(like most Linux libraries).
This means that the hash has to change _only_  if the major version of
the dependency changes (you need to recompile your "restricted" code
anyway in this case), or if the configuration of the dependency
changes in an incompatible way (ouch!).
Of course this is only true for library dependencies.

I am not sure if it is possible to just "configure" the hashing
algorithm of bitbake to consider only ABI / API breakage instead of
"always rehashing every change recursively".
Some other thoughts: It may be possible to inject packages during the
build via binary package feed...

Both ideas would require expanding the bitbake code though.

2013/3/27 Jerrod Peach <peachj@lexmark.com>:
> Mark,
>
> Option 1 isn't a very big concern since it's likely only developers working
> on the restricted code will cause its hashes to permute.
>
> Option 2 is precisely what we were thinking about doing.  The concern,
> though, is what happens when a developer who doesn't have access to the code
> changes a dependency of the restricted code in such a way that it actually
> affects the restricted code.  That developer is unable to test the effects
> of his changes without pushing them to a back-end build, which would involve
> pushing untested changes somewhere.  I don't love that thought.
>
> Another option would be to make the recipe smart enough to build restricted
> source on a different machine from the user's using an SDK (which is what I
> think you were suggesting), but that involves transferring the user's
> uncommitted code and/or built objects to the separate machine, adding a
> fairly significant layer of complexity (and likely a significant amount of
> time) to what is otherwise a relatively straightforward build process.
>
> I'm not the biggest fan of any of these options presently, but if we have to
> choose one of these routes, we will.  (We were likely going to use what you
> presented as your second option, but that's still going to involve someone
> updating recipes for restricted source every time new binaries become
> available.)  I do very much appreciate the input and I'm glad to know the
> ideas we came up with aren't completely crazy, since someone else has
> thought of them too ;-)  Any other thoughts on this issue?
>
> Kind regards,
>
> Jerrod
>
>
> On Tue, Mar 26, 2013 at 3:21 PM, Mark Hatle <mark.hatle@windriver.com>
> wrote:
>>
>> On 3/26/13 7:53 AM, Jerrod Peach wrote:
>>>
>>> As part of my company's firmware builds, we have to build some code that
>>> only a
>>> handful of developers are allowed to view.  We call this restricted
>>> source code.
>>>   Getting our official "system" builds to build this code isn't a
>>> problem.  What
>>> is a problem is a regular developer's build of this code.
>>>
>>> Imagine this scenario: The restricted source depends upon eglibc.  Our
>>> low-level
>>> team, who doesn't have access to the restricted source, updates the
>>> recipe for
>>> eglibc.  The hash for the restricted package is permuted, and so they
>>> can't get
>>> an sstate hit and are required to rebuild the source.  But, since they
>>> can't
>>> check out the source, they can't build it.  This would cause a build
>>> failure.
>>>
>>> Is anyone dealing with this scenario while using Yocto currently?  If so,
>>> how?
>>>   I know it may be unlikely that many people are hitting this since Yocto
>>> is
>>> primarily used to build open-source code, but I thought I'd take a shot
>>> in the
>>> dark and hope for the best.  :-)
>>>
>>
>> Two ways I know of doing this.  Slightly different way of doing things:
>>
>> 1) If the code does -not- rely on outside influences, such as eglibc.
>> Configure the recipe and pretty much ignore everything else with a
>> vardepsexp.  Then ship the sstate-cache files that cover the compiled code.
>> (Along with the original recipe...)
>>
>> 2) For code that DOES rely on outside influences.. create a custom recipe
>> that either downloads the source and builds it [if the user has access to
>> the source], or will pull the binaries from a specific location and simply
>> install/package them.   This is actually the more common approach.
>>
>> (To seed that location, you can extract the items from your restricted
>> build -- or build it outside the tree using an SDK or similar.)
>>
>> --Mark
>>
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>>>
>>
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
Regards
Samuel