How to put a correct dependency with regards to gcc?

How to put a correct dependency with regards to gcc?

[Reshetova, Elena]

[-- Attachment #1.1: Type: text/plain, Size: 1212 bytes --]

Hi,

 

I have a simple problem but not able to come with simple and elegant
solution for it, so would be very thankful for the suggestions. 

 

I have a task that is specified to run after do_fetch. The task needs the
source to be *actually* fetched, since it operates on archives. It works
well for everything apart gcc. 

For example in fido branch, my task run for libgcc will fail, because
do_fetch task for libgcc actually *does not* not fetch the gcc sources (only
puts a nice lock file in the download folder).

 

When I noticed this, I tried to put the dependency on gcc-source:do_fetch
for my task (because that actually fetches the gcc sources) and it works in
fido, but in master after changes to gcc nothing provides gcc-source and it
wants to specify the version, which I don't want to do (ideally I don't want
to put any dependencies to gcc, since I don't really depend on it!). In
master a run for libgcc-initial fails for the same reasons: gcc tar ball is
not there. 

 

Is there a way to make it work correctly? 

 

Here is the history of issue we are trying to solve:
https://github.com/01org/meta-security-isafw/issues/1

 

Best Regards,
Elena.


[-- Attachment #1.2: Type: text/html, Size: 3733 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7586 bytes --]

Re: How to put a correct dependency with regards to gcc?

[Randy MacLeod]

On 2015-09-04 01:03 PM, Reshetova, Elena wrote:
> Hi,
>
> I have a simple problem but not able to come with simple and elegant
> solution for it, so would be very thankful for the suggestions.
>
> I have a task that is specified to run after do_fetch. The task needs
> the source to be **actually** fetched, since it operates on archives. It
> works well for everything apart gcc.
>
> For example in fido branch, my task run for libgcc will fail, because
> do_fetch task for libgcc actually **does not** not fetch the gcc sources
> (only puts a nice lock file in the download folder).
>
> When I noticed this, I tried to put the dependency on
> gcc-source:do_fetch for my task (because that actually fetches the gcc
> sources) and it works in fido, but in master after changes to gcc
> nothing provides gcc-source and it wants to specify the version, which I
> don’t want to do (ideally I don’t want to put any dependencies to gcc,
> since I don’t really depend on it!). In master a run for libgcc-initial
> fails for the same reasons: gcc tar ball is not there.
>
> Is there a way to make it work correctly?

Maybe we have to fix gcc-source after all rather than use the
EXCLUDE_WORLD change I made in:
    https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878
Does your scheme work for the kernel source? See:

Search for "kernel-devsrc" here:
  http://comments.gmane.org/gmane.comp.handhelds.openembedded.core/68352

../Randy

>
> Here is the history of issue we are trying to solve:
> https://github.com/01org/meta-security-isafw/issues/1
>
> Best Regards,
> Elena.
>
>
>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5

Re: How to put a correct dependency with regards to gcc?

[Patrick Ohly]

On Tue, 2015-09-08 at 22:45 -0400, Randy MacLeod wrote:
> Maybe we have to fix gcc-source after all rather than use the
> EXCLUDE_WORLD change I made in:
>     https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878
> Does your scheme work for the kernel source? See:
> 
> Search for "kernel-devsrc" here:
>   http://comments.gmane.org/gmane.comp.handhelds.openembedded.core/68352

Also possibly related: 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7959

"archiver.bbclass: fails for linux--yocto when archiving patched source"

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: How to put a correct dependency with regards to gcc?

[Reshetova, Elena]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

On Tue, 2015-09-08 at 22:45 -0400, Randy MacLeod wrote:
> Maybe we have to fix gcc-source after all rather than use the
> EXCLUDE_WORLD change I made in:
>     https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878
> Does your scheme work for the kernel source? See:
>
> Search for "kernel-devsrc" here:
>
> http://comments.gmane.org/gmane.comp.handhelds.openembedded.core/68352

Also possibly related:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7959

"archiver.bbclass: fails for linux--yocto when archiving patched source"

Not sure it is 100% related since my issue with doing unpacking is that 
archive is simply not present *at all* in the download dir. So for me it seems 
some kind of dependency issue. If I execute do_fetch on gcc, it should put a 
dependency to gcc-source do_fetch if this is smth that  actually would 
download the sources.

Best Regards,
Elena.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7586 bytes --]

Re: How to put a correct dependency with regards to gcc?

[Reshetova, Elena]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

>Maybe we have to fix gcc-source after all rather than use the EXCLUDE_WORLD
change I made in:
>    https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878

Should I file a bug about this for easy tracking? My problem really seems
that do_fetch on some related to gcc packages doesn't result in fetch of
sources. 

>Does your scheme work for the kernel source? See:

>Search for "kernel-devsrc" here:
>  http://comments.gmane.org/gmane.comp.handhelds.openembedded.core/68352

There doesn't seem to be an issue with kernel source packages for my task. I
think this is because the sources are correctly fetched by dependent
packages.

Best Regards,
Elena.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7586 bytes --]

Re: How to put a correct dependency with regards to gcc?

[Randy MacLeod]

On 2015-09-09 12:09 PM, Reshetova, Elena wrote:
>> Maybe we have to fix gcc-source after all rather than use the EXCLUDE_WORLD
> change I made in:
>>     https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878
>
> Should I file a bug about this for easy tracking? My problem really seems
> that do_fetch on some related to gcc packages doesn't result in fetch of
> sources.

I'll send a fix early this week.

../Randy

>
>> Does your scheme work for the kernel source? See:
>
>> Search for "kernel-devsrc" here:
>>   http://comments.gmane.org/gmane.comp.handhelds.openembedded.core/68352
>
> There doesn't seem to be an issue with kernel source packages for my task. I
> think this is because the sources are correctly fetched by dependent
> packages.
>
> Best Regards,
> Elena.
>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5

Re: How to put a correct dependency with regards to gcc?

[Reshetova, Elena]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]


On 2015-09-09 12:09 PM, Reshetova, Elena wrote:
>> Maybe we have to fix gcc-source after all rather than use the 
>> EXCLUDE_WORLD
> change I made in:
>>     https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878
>
> Should I file a bug about this for easy tracking? My problem really 
> seems that do_fetch on some related to gcc packages doesn't result in 
> fetch of sources.

I'll send a fix early this week.


Thank you very much! Greatly appreciated!

Best Regards,
Elena.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7586 bytes --]

Re: How to put a correct dependency with regards to gcc?

[Randy MacLeod]

On 2015-09-14 01:55 PM, Randy MacLeod wrote:
> On 2015-09-09 12:09 PM, Reshetova, Elena wrote:
>>> Maybe we have to fix gcc-source after all rather than use the
>>> EXCLUDE_WORLD
>> change I made in:
>>>     https://bugzilla.yoctoproject.org/show_bug.cgi?id=7878
>>
>> Should I file a bug about this for easy tracking? My problem really seems
>> that do_fetch on some related to gcc packages doesn't result in fetch of
>> sources.
>
> I'll send a fix early this week.
>
> ../Randy
>
>>
>>> Does your scheme work for the kernel source? See:
>>
>>> Search for "kernel-devsrc" here:
>>>   http://comments.gmane.org/gmane.comp.handhelds.openembedded.core/68352
>>
>> There doesn't seem to be an issue with kernel source packages for my
>> task. I
>> think this is because the sources are correctly fetched by dependent
>> packages.
>>
>> Best Regards,
>> Elena.
>>
>

I haven't been able to come up with a scheme that works yet.
With the patch below, I get:

I'll need someone to explain the intent of the gcc-* pkgs
design or more time to dig though the files and history.

Robert tells me that my idea that I need a bitbake rule to:
  " call the parent implementation or
    if there isn't one, return success."
has been discussed before and it is not easy to do.

For me, this is "a nice to have" feature that could wait
for oe-core-2.1.

../Randy



diff --git a/meta/recipes-devtools/gcc/gcc-sanitizers.inc 
b/meta/recipes-devtools/gcc/gcc-sanitizers.inc
index c987ccb..1da7bda 100644
--- a/meta/recipes-devtools/gcc/gcc-sanitizers.inc
+++ b/meta/recipes-devtools/gcc/gcc-sanitizers.inc
@@ -5,12 +5,18 @@ LICENSE = "NCSA | MIT"
  LIC_FILES_CHKSUM = "\
      file://libsanitizer/LICENSE.TXT;md5=0249c37748936faf5b1efd5789587909 \
  "
+EXCLUDE_FROM_WORLD = "1"

  EXTRA_OECONF_PATHS = "\
      --with-sysroot=/not/exist \
      --with-build-sysroot=${STAGING_DIR_TARGET} \
  "

+# These stages are done by the sister gcc package:
+# do_unpack[noexec] = "1"
+# do_patch[noexec] = "1"
+
+
  do_configure () {
      mtarget=`echo ${TARGET_SYS} | sed -e s#-${SDKPKGSUFFIX}##`
      target=`echo ${TARGET_SYS} | sed -e s#-${SDKPKGSUFFIX}##`
diff --git a/meta/recipes-devtools/gcc/gcc-source.inc 
b/meta/recipes-devtools/gcc/gcc-source.inc
index 794fd4d..ef4ff60 100644
--- a/meta/recipes-devtools/gcc/gcc-source.inc
+++ b/meta/recipes-devtools/gcc/gcc-source.inc
@@ -1,15 +1,16 @@
-deltask do_configure
-deltask do_compile
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_install
-deltask do_populate_sysroot
-deltask do_populate_lic
-deltask do_package_qa
-deltask do_packagedata
-deltask do_rm_work
+# gcc-source should only be fetched, unpacked and patched.
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_package[noexec] = "1"
+do_package_write_rpm[noexec] = "1"
+do_package_write_ipk[noexec] = "1"
+do_package_write_deb[noexec] = "1"
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
+do_populate_lic[noexec] = "1"
+do_package_qa[noexec] = "1"
+do_packagedata[noexec] = "1"
+do_rm_work[noexec] = "1"

  PN = "gcc-source-${PV}"
  WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5

Re: How to put a correct dependency with regards to gcc?

[Richard Purdie]

On Tue, 2015-09-15 at 22:37 -0400, Randy MacLeod wrote:
> I haven't been able to come up with a scheme that works yet.
> With the patch below, I get:
> 
> I'll need someone to explain the intent of the gcc-* pkgs
> design or more time to dig though the files and history.
> 
> Robert tells me that my idea that I need a bitbake rule to:
>   " call the parent implementation or
>     if there isn't one, return success."
> has been discussed before and it is not easy to do.
> 
> For me, this is "a nice to have" feature that could wait
> for oe-core-2.1.

The idea is quite simple. Rather than having a copy of the gcc source
for each recipe variant (-cross-initial, -cross, -crosssdk-initial,
-crosssdk, -cross-canadian etc.) we have a single copy of the source.

We tried an older shared stamp scheme which was fragile and prone to
weird failures.  Instead we created the gcc-source recipe which is
responsible for the fetch/unpack/patch/preconfigure and then each recipe
can work off the shared source (and has a dependency on gcc-source).

For Elena's use case, I therefore think it might be better to analyse
the shared source once and not in the case of each recipe (e.g. if
SRC_URI is empty).

Cheers,

Richard

Re: How to put a correct dependency with regards to gcc?

[Reshetova, Elena]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

> The idea is quite simple. Rather than having a copy of the gcc source for 
> each recipe variant 
> (-cross-initial, -cross, -crosssdk-initial, -crosssdk, -cross-canadian etc.) 
> we have a single copy of the source.

>We tried an older shared stamp scheme which was fragile and prone to weird 
>failures.  Instead we created the gcc-source recipe which is responsible for 
>the fetch/unpack/patch/preconfigure and then each recipe can work off the 
>shared source (and has >a dependency on gcc-source).

Thank you Richard for explaining this! It helps greatly!

>For Elena's use case, I therefore think it might be better to analyse the 
>shared source once and not in the case of each recipe (e.g. if SRC_URI is 
>empty).

I am really not sure how to do it apart from hardcoding a check inside the 
class that if the recipe happens to be gcc related, run the analysis only once 
for gcc-source and do not run it for other related packages.
My task is currently run for every recipe uniformly and it would be not that 
elegant to have this hardcoding part. Is there a better way of handling it?

Best Regards,
Elena.





[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7586 bytes --]

Re: How to put a correct dependency with regards to gcc?

[Richard Purdie]

On Tue, 2015-09-22 at 11:03 +0000, Reshetova, Elena wrote:
> > The idea is quite simple. Rather than having a copy of the gcc source for 
> > each recipe variant 
> > (-cross-initial, -cross, -crosssdk-initial, -crosssdk, -cross-canadian etc.) 
> > we have a single copy of the source.
> 
> >We tried an older shared stamp scheme which was fragile and prone to weird 
> >failures.  Instead we created the gcc-source recipe which is responsible for 
> >the fetch/unpack/patch/preconfigure and then each recipe can work off the 
> >shared source (and has >a dependency on gcc-source).
> 
> Thank you Richard for explaining this! It helps greatly!
> 
> >For Elena's use case, I therefore think it might be better to analyse the 
> >shared source once and not in the case of each recipe (e.g. if SRC_URI is 
> >empty).
> 
> I am really not sure how to do it apart from hardcoding a check inside the 
> class that if the recipe happens to be gcc related, run the analysis only once 
> for gcc-source and do not run it for other related packages.
> My task is currently run for every recipe uniformly and it would be not that 
> elegant to have this hardcoding part. Is there a better way of handling it?

You could test if SRC_URI is empty in your code?

Then all we'd need to do is ensure that SRC_URI was empty for the gcc
recipes except for gcc-source?

Cheers,

Richard

Re: How to put a correct dependency with regards to gcc?

[Reshetova, Elena]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]

On Tue, 2015-09-22 at 11:03 +0000, Reshetova, Elena wrote:
> > The idea is quite simple. Rather than having a copy of the gcc
> > source for each recipe variant (-cross-initial, -cross,
> > -crosssdk-initial, -crosssdk, -cross-canadian etc.) we have a single
> > copy of the source.
>
> >We tried an older shared stamp scheme which was fragile and prone to
> >weird failures.  Instead we created the gcc-source recipe which is
> >responsible for the fetch/unpack/patch/preconfigure and then each
> >recipe can work off the shared source (and has >a dependency on 
> >gcc-source).
>
> Thank you Richard for explaining this! It helps greatly!
>
> >For Elena's use case, I therefore think it might be better to analyse
> >the shared source once and not in the case of each recipe (e.g. if
> >SRC_URI is empty).
>
> I am really not sure how to do it apart from hardcoding a check inside
> the class that if the recipe happens to be gcc related, run the
> analysis only once for gcc-source and do not run it for other related 
> packages.
> My task is currently run for every recipe uniformly and it would be
> not that elegant to have this hardcoding part. Is there a better way of 
> handling it?

>You could test if SRC_URI is empty in your code?

>Then all we'd need to do is ensure that SRC_URI was empty for the gcc recipes 
>except for gcc-source?

Yes, I think this would work. There is already a check: 
https://github.com/01org/meta-security-isafw/blob/master/classes/isafw.bbclass#L39 
but currently it does return "True" for things like libgcc and the task fails 
since there are no actual sources.

Best Regards,
Elena.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 7586 bytes --]

Re: How to put a correct dependency with regards to gcc?

[Richard Purdie]

On Fri, 2015-09-04 at 17:03 +0000, Reshetova, Elena wrote:
> I have a simple problem but not able to come with simple and elegant
> solution for it, so would be very thankful for the suggestions. 
>  
> I have a task that is specified to run after do_fetch. The task needs
> the source to be *actually* fetched, since it operates on archives. It
> works well for everything apart gcc. 
>
> For example in fido branch, my task run for libgcc will fail, because
> do_fetch task for libgcc actually *does not* not fetch the gcc sources
> (only puts a nice lock file in the download folder).
>  
> When I noticed this, I tried to put the dependency on
> gcc-source:do_fetch for my task (because that actually fetches the gcc
> sources) and it works in fido, but in master after changes to gcc
> nothing provides gcc-source and it wants to specify the version, which
> I don’t want to do (ideally I don’t want to put any dependencies to
> gcc, since I don’t really depend on it!). In master a run for
> libgcc-initial fails for the same reasons: gcc tar ball is not there. 
>  
> Is there a way to make it work correctly? 

We could change gcc-shared-work.inc to add:

do_fetch[depends] += "gcc-source-${PV}:do_fetch"

which I think might fix the issues you're seeing. The good news is that
gcc is pretty unique in this regard so you shouldn't see the issue
anywhere else.

I'm also wondering whether we simply should set SRC_URI to be empty
here. The source is fetched, unpacked and patched as part of the
gcc-source recipe. Your code should only really analyse the copy in
gcc-source, its the same source used by all the different gcc variants.

Cheers,

Richard