[morty/rocko/master][PATCH] ti-pdk: Weakly assign PARALLEL_XDC

[morty/rocko/master][PATCH] ti-pdk: Weakly assign PARALLEL_XDC

[Joshua Watt]

Allowing it to be overridden in local.conf

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 classes/ti-pdk.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
index 540a665c..0b0c9ffe 100644
--- a/classes/ti-pdk.bbclass
+++ b/classes/ti-pdk.bbclass
@@ -63,7 +63,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
 
 TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
 
-PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
+PARALLEL_XDC ?= "--jobs=${BB_NUMBER_THREADS}"
 PARALLEL_MAKE = ""
 
 def get_doxygen_support(d):
-- 
2.14.3

Re: [morty/rocko/master][PATCH] ti-pdk: Weakly assign PARALLEL_XDC

[Denys Dmytriyenko]

On Mon, Apr 02, 2018 at 02:47:28PM -0500, Joshua Watt wrote:
> Allowing it to be overridden in local.conf

Why?


> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  classes/ti-pdk.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
> index 540a665c..0b0c9ffe 100644
> --- a/classes/ti-pdk.bbclass
> +++ b/classes/ti-pdk.bbclass
> @@ -63,7 +63,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
>  
>  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
>  
> -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
> +PARALLEL_XDC ?= "--jobs=${BB_NUMBER_THREADS}"
>  PARALLEL_MAKE = ""
>  
>  def get_doxygen_support(d):
> -- 
> 2.14.3
> 
> -- 
> _______________________________________________
> meta-ti mailing list
> meta-ti@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-ti

Re: [morty/rocko/master][PATCH] ti-pdk: Weakly assign PARALLEL_XDC

[Joshua Watt]

On Mon, 2018-04-02 at 16:20 -0400, Denys Dmytriyenko wrote:
> On Mon, Apr 02, 2018 at 02:47:28PM -0500, Joshua Watt wrote:
> > Allowing it to be overridden in local.conf
> 
> Why?

For some reason, when some ti-pdk recipes are built on my machine, the
number of threads explodes exponentially.... It ends up launching
several hundreds to thousands of threads which consequently make my
computer unresponsive for quite a while, and I usually have to reboot
it afterwards because something important stopped working.

I think for some reason the --jobs= flag is getting compounded
recursively.

Admittedly, there appears to be some deeper bug with the job
parallelism that needs to be looked into, but this at least allows me
to continue working (by setting PARALLEL_XDC = "" in local.conf) in the
meantime.

I'm on the latest HEAD of morty BTW.

> 
> 
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> >  classes/ti-pdk.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
> > index 540a665c..0b0c9ffe 100644
> > --- a/classes/ti-pdk.bbclass
> > +++ b/classes/ti-pdk.bbclass
> > @@ -63,7 +63,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
> >  
> >  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
> >  
> > -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
> > +PARALLEL_XDC ?= "--jobs=${BB_NUMBER_THREADS}"
> >  PARALLEL_MAKE = ""
> >  
> >  def get_doxygen_support(d):
> > -- 
> > 2.14.3
> > 
> > -- 
> > _______________________________________________
> > meta-ti mailing list
> > meta-ti@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/meta-ti

Re: [morty/rocko/master][PATCH] ti-pdk: Weakly assign PARALLEL_XDC

[Denys Dmytriyenko]

On Mon, Apr 02, 2018 at 04:00:24PM -0500, Joshua Watt wrote:
> On Mon, 2018-04-02 at 16:20 -0400, Denys Dmytriyenko wrote:
> > On Mon, Apr 02, 2018 at 02:47:28PM -0500, Joshua Watt wrote:
> > > Allowing it to be overridden in local.conf
> > 
> > Why?
> 
> For some reason, when some ti-pdk recipes are built on my machine, the
> number of threads explodes exponentially.... It ends up launching
> several hundreds to thousands of threads which consequently make my
> computer unresponsive for quite a while, and I usually have to reboot
> it afterwards because something important stopped working.
> 
> I think for some reason the --jobs= flag is getting compounded
> recursively.
> 
> Admittedly, there appears to be some deeper bug with the job
> parallelism that needs to be looked into, but this at least allows me
> to continue working (by setting PARALLEL_XDC = "" in local.conf) in the
> meantime.
> 
> I'm on the latest HEAD of morty BTW.

Do you only see this behavior on morty? What about rocko/master?

I've asked around and we don't see this issue internally and we do a lot of 
builds in our build farm on a daily basis...

Since this is otherwise a harmless change, I don't mind merging it. Please let 
me know if you actually do need it in all 3 branches. Thanks.


> > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > ---
> > >  classes/ti-pdk.bbclass | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
> > > index 540a665c..0b0c9ffe 100644
> > > --- a/classes/ti-pdk.bbclass
> > > +++ b/classes/ti-pdk.bbclass
> > > @@ -63,7 +63,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
> > >  
> > >  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
> > >  
> > > -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
> > > +PARALLEL_XDC ?= "--jobs=${BB_NUMBER_THREADS}"
> > >  PARALLEL_MAKE = ""
> > >  
> > >  def get_doxygen_support(d):
> > > -- 
> > > 2.14.3
> > > 
> > > -- 
> > > _______________________________________________
> > > meta-ti mailing list
> > > meta-ti@yoctoproject.org
> > > https://lists.yoctoproject.org/listinfo/meta-ti

Re: [morty/rocko/master][PATCH] ti-pdk: Weakly assign PARALLEL_XDC

[Joshua Watt]

On Thu, 2018-04-12 at 12:51 -0400, Denys Dmytriyenko wrote:
> On Mon, Apr 02, 2018 at 04:00:24PM -0500, Joshua Watt wrote:
> > On Mon, 2018-04-02 at 16:20 -0400, Denys Dmytriyenko wrote:
> > > On Mon, Apr 02, 2018 at 02:47:28PM -0500, Joshua Watt wrote:
> > > > Allowing it to be overridden in local.conf
> > > 
> > > Why?
> > 
> > For some reason, when some ti-pdk recipes are built on my machine,
> > the
> > number of threads explodes exponentially.... It ends up launching
> > several hundreds to thousands of threads which consequently make my
> > computer unresponsive for quite a while, and I usually have to
> > reboot
> > it afterwards because something important stopped working.
> > 
> > I think for some reason the --jobs= flag is getting compounded
> > recursively.
> > 
> > Admittedly, there appears to be some deeper bug with the job
> > parallelism that needs to be looked into, but this at least allows
> > me
> > to continue working (by setting PARALLEL_XDC = "" in local.conf) in
> > the
> > meantime.
> > 
> > I'm on the latest HEAD of morty BTW.
> 
> Do you only see this behavior on morty? What about rocko/master?
> 
> I've asked around and we don't see this issue internally and we do a
> lot of 
> builds in our build farm on a daily basis...
> 
> Since this is otherwise a harmless change, I don't mind merging it.
> Please let 
> me know if you actually do need it in all 3 branches. Thanks.

Ya, sorry. I've been meaning to loop back around on this one. I was
able to track this down a bit further to the ti-ipc-rtos recipe. For
some reason, that specific recipe behaves very poorly on my machine,
launching thousands of threads. I'm not sure where to go about
debugging that if you happen to have any hints, I can look into it.

For some reason, the ti-ipc-rtos recipe doesn't respect PARALLEL_XDC
like all the other recipes, as it manually adds:

 JOBS="--jobs=${BB_NUMBER_THREADS}"

to EXTRA_OEMAKE. I had to make an additional local change:

 JOBS="${PARALLEL_XDC}"


If that can be sorted out, this patch is probably not necessary on all
branches at the current time. It does seem like behavior that would be
nice to control, but I think it is also incomplete at this point
because you probably want the change described above, and also want to
add:

 BB_HASHBASE_WHITELIST_append = " PARALLEL_XDC"

to layer.conf to prevent changes to the variable from causing
unnecessary rebuilds.

I'll push up a new patchset with these changes and you can decide if
you want it on master. Any insight (or debugging tips) as to why ti-
ipc-rtos behaves so badly (on morty at least) would be greatly
appreciated.

Thanks,
Joshua Watt
 
> 
> 
> > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > > ---
> > > >  classes/ti-pdk.bbclass | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/classes/ti-pdk.bbclass b/classes/ti-pdk.bbclass
> > > > index 540a665c..0b0c9ffe 100644
> > > > --- a/classes/ti-pdk.bbclass
> > > > +++ b/classes/ti-pdk.bbclass
> > > > @@ -63,7 +63,7 @@ TI_PDK_DOXYGEN_SUPPORT ?= "1"
> > > >  
> > > >  TI_PDK_XDC_ARGS ?= "${TI_PDK_LIMIT_SOCS}"
> > > >  
> > > > -PARALLEL_XDC = "--jobs=${BB_NUMBER_THREADS}"
> > > > +PARALLEL_XDC ?= "--jobs=${BB_NUMBER_THREADS}"
> > > >  PARALLEL_MAKE = ""
> > > >  
> > > >  def get_doxygen_support(d):
> > > > -- 
> > > > 2.14.3
> > > > 
> > > > -- 
> > > > _______________________________________________
> > > > meta-ti mailing list
> > > > meta-ti@yoctoproject.org
> > > > https://lists.yoctoproject.org/listinfo/meta-ti