Deployment for machine X will remove its results from machine Y's deploy dir

Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" 
into a recipe dir. Then build it for two machines. Building it for one machine 
will remove it from the deployment directory of the other. This problem has 
been bugging me for months, I had files just "disappear" mysteriously from the 
deploy directory and seemingly random times, and now I finally figured out 
what causes it.

(cut here)

SUMMARY = "Demonstrate a bug in OE deployment"
DESCRIPTION = "Build this package for a machine X, then look at the image's \
  deploy directory. You'll see a deployme.txt there. Now build it for another \
  machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
  from the image dir. This appears to be a bug in OE's deployment."
LICENSE = "GPLv2"
LIC_FILES_CHKSUM = 
"file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"

inherit allarch deploy

do_compile () {
	echo "Hello world!" > deployme.txt
}

do_deploy () {
     install -d ${DEPLOYDIR}
     install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
}

addtask deploy before do_build after do_compile

(cut here)


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Gary Thomas]

On 2014-11-27 01:35, Mike Looijmans wrote:
> Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
> deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
> now I finally figured out what causes it.
>
> (cut here)
>
> SUMMARY = "Demonstrate a bug in OE deployment"
> DESCRIPTION = "Build this package for a machine X, then look at the image's \
>   deploy directory. You'll see a deployme.txt there. Now build it for another \
>   machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
>   from the image dir. This appears to be a bug in OE's deployment."
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
>
> inherit allarch deploy
>
> do_compile () {
>      echo "Hello world!" > deployme.txt
> }
>
> do_deploy () {
>      install -d ${DEPLOYDIR}
>      install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
> }
>
> addtask deploy before do_build after do_compile
>
> (cut here)

Very interesting & verified with the latest master.

Have you filed a bug?  https://bugzilla.yoctoproject.org/

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Richard Purdie]

On Thu, 2014-11-27 at 05:02 -0700, Gary Thomas wrote:
> On 2014-11-27 01:35, Mike Looijmans wrote:
> > Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
> > deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
> > now I finally figured out what causes it.
> >
> > (cut here)
> >
> > SUMMARY = "Demonstrate a bug in OE deployment"
> > DESCRIPTION = "Build this package for a machine X, then look at the image's \
> >   deploy directory. You'll see a deployme.txt there. Now build it for another \
> >   machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
> >   from the image dir. This appears to be a bug in OE's deployment."
> > LICENSE = "GPLv2"
> > LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
> >
> > inherit allarch deploy
> >
> > do_compile () {
> >      echo "Hello world!" > deployme.txt
> > }
> >
> > do_deploy () {
> >      install -d ${DEPLOYDIR}
> >      install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
> > }
> >
> > addtask deploy before do_build after do_compile
> >
> > (cut here)
> 
> Very interesting & verified with the latest master.
> 
> Have you filed a bug?  https://bugzilla.yoctoproject.org/

Well, I'm not convinced this is a bug as such. You've created an
"allarch" deploy task, how would you expect this to behave?

"allarch" means that the output from this task is universal and can be
used on all targets. It will therefore get run once.

A "deploy" task is machine specific.

What ends up happening is therefore the task has a stamp is
"universally" created. When you change machine, the checksum of the task
changes, the previous version is removed, the new version is installed.

So in many ways the system is doing exactly what I would expect it to do
and it isn't a bug in that sense.

The real question is how should an "allarch" + "deploy" task behave when
you've specified machine specific paths? Perhaps erroring would be
better?

Cheers,

Richard

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

On 11/27/2014 02:17 PM, Richard Purdie wrote:
> On Thu, 2014-11-27 at 05:02 -0700, Gary Thomas wrote:
>> On 2014-11-27 01:35, Mike Looijmans wrote:
>>> Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
>>> deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
>>> now I finally figured out what causes it.
>>>
>>> (cut here)
>>>
>>> SUMMARY = "Demonstrate a bug in OE deployment"
>>> DESCRIPTION = "Build this package for a machine X, then look at the image's \
>>>    deploy directory. You'll see a deployme.txt there. Now build it for another \
>>>    machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
>>>    from the image dir. This appears to be a bug in OE's deployment."
>>> LICENSE = "GPLv2"
>>> LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
>>>
>>> inherit allarch deploy
>>>
>>> do_compile () {
>>>       echo "Hello world!" > deployme.txt
>>> }
>>>
>>> do_deploy () {
>>>       install -d ${DEPLOYDIR}
>>>       install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
>>> }
>>>
>>> addtask deploy before do_build after do_compile
>>>
>>> (cut here)
>>
>> Very interesting & verified with the latest master.
>>
>> Have you filed a bug?  https://bugzilla.yoctoproject.org/
>
> Well, I'm not convinced this is a bug as such. You've created an
> "allarch" deploy task, how would you expect this to behave?
>
> "allarch" means that the output from this task is universal and can be
> used on all targets. It will therefore get run once.
>
> A "deploy" task is machine specific.
>
> What ends up happening is therefore the task has a stamp is
> "universally" created. When you change machine, the checksum of the task
> changes, the previous version is removed, the new version is installed.
>
> So in many ways the system is doing exactly what I would expect it to do
> and it isn't a bug in that sense.

It's not a bug in the sense that it doesn't do as it was programmed to do. I 
understand what's happening here.

I just think that the logic here is wrong. If "deploy" is machine specific, 
then the implicit "undeploy" should be machine specific too, right?

> The real question is how should an "allarch" + "deploy" task behave when
> you've specified machine specific paths? Perhaps erroring would be
> better?

That would mean that roughly all deploy tasks will fail. At best they're tied 
to MACHINE_ARCH, but never to MACHINE itself.

Would be strange to put PACKAGE_ARCH="${MACHINE}" in a recipe that clearly has 
no dependency on machine specific things. And I wrote "${MACHINE}" here on 
purpose.

I was thinking along the lines of "DEPLOY_DIR_IMAGE must have the same prefix" 
or so.

If I knew the solution, I'd have posted a patch instead of a question or report.

M.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Richard Purdie]

On Thu, 2014-11-27 at 15:22 +0100, Mike Looijmans wrote:
> On 11/27/2014 02:17 PM, Richard Purdie wrote:
> > On Thu, 2014-11-27 at 05:02 -0700, Gary Thomas wrote:
> >> On 2014-11-27 01:35, Mike Looijmans wrote:
> >>> Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
> >>> deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
> >>> now I finally figured out what causes it.
> >>>
> >>> (cut here)
> >>>
> >>> SUMMARY = "Demonstrate a bug in OE deployment"
> >>> DESCRIPTION = "Build this package for a machine X, then look at the image's \
> >>>    deploy directory. You'll see a deployme.txt there. Now build it for another \
> >>>    machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
> >>>    from the image dir. This appears to be a bug in OE's deployment."
> >>> LICENSE = "GPLv2"
> >>> LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
> >>>
> >>> inherit allarch deploy
> >>>
> >>> do_compile () {
> >>>       echo "Hello world!" > deployme.txt
> >>> }
> >>>
> >>> do_deploy () {
> >>>       install -d ${DEPLOYDIR}
> >>>       install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
> >>> }
> >>>
> >>> addtask deploy before do_build after do_compile
> >>>
> >>> (cut here)
> >>
> >> Very interesting & verified with the latest master.
> >>
> >> Have you filed a bug?  https://bugzilla.yoctoproject.org/
> >
> > Well, I'm not convinced this is a bug as such. You've created an
> > "allarch" deploy task, how would you expect this to behave?
> >
> > "allarch" means that the output from this task is universal and can be
> > used on all targets. It will therefore get run once.
> >
> > A "deploy" task is machine specific.
> >
> > What ends up happening is therefore the task has a stamp is
> > "universally" created. When you change machine, the checksum of the task
> > changes, the previous version is removed, the new version is installed.
> >
> > So in many ways the system is doing exactly what I would expect it to do
> > and it isn't a bug in that sense.
> 
> It's not a bug in the sense that it doesn't do as it was programmed to do.

Its doing *exactly* what the was designed to do. That doesn't match what
you want/expect though.

>  I understand what's happening here.
> 
> I just think that the logic here is wrong. If "deploy" is machine specific, 
> then the implicit "undeploy" should be machine specific too, right?

Well, its more complex than that.

deploy.bbclass defaults to DEPLOY_DIR_IMAGE.

DEPLOY_DIR_IMAGE defaults to ${DEPLOY_DIR}/images/${MACHINE} 

I actually put off merging the latter since I knew it would cause
issues, I just couldn't articulate all of them at the time :(.

> > The real question is how should an "allarch" + "deploy" task behave when
> > you've specified machine specific paths? Perhaps erroring would be
> > better?
> 
> That would mean that roughly all deploy tasks will fail.

I'm not sure we have many deploy+allarch tasks so I think "roughly none"
would be a better description.

deploy is usually used for bootloaders and kernels, both of which are
not allarch.

> At best they're tied 
> to MACHINE_ARCH, but never to MACHINE itself.

No, they're tried to MACHINE itself, see above.

> Would be strange to put PACKAGE_ARCH="${MACHINE}" in a recipe that clearly has 
> no dependency on machine specific things. And I wrote "${MACHINE}" here on 
> purpose.

Dropping the "allarch" would be better than that.

> I was thinking along the lines of "DEPLOY_DIR_IMAGE must have the same prefix" 
> or so.
> 
> If I knew the solution, I'd have posted a patch instead of a question or report.

Well, allarch.bbclass could override DEPLOY_DIR_IMAGE to remove
the /${MACHINE} suffix. That would appear to fix the issues you're
seeing, at the risk of having a different group of people upset that you
don't get a complete directory per machine.

It comes down to which behaviour we want. Changing MACHINE in the
definition of DEPLOY_DIR_IMAGE to PACKAGE_ARCH might be the better
solution, then it will deploy based on how specific or not specific the
resulting code it. That will likely upset certain people even more
though. The other option is to accept that its machine specific and do
PACKAGE_ARCH="${MACHINE_ARCH}" in the class. That is suboptimal for the
reasons you describe but would get the behaviour some people want.

I suspect there isn't a right answer here :(

An opinion from Martin/Koen might be useful at this point since it could
affect them more than others (as well as things like the yocto project
autobuilder output and output processing/testing/releasing).

Cheers,

Richard

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

On 27-11-2014 15:41, Richard Purdie wrote:
> On Thu, 2014-11-27 at 15:22 +0100, Mike Looijmans wrote:
>> On 11/27/2014 02:17 PM, Richard Purdie wrote:
>>> On Thu, 2014-11-27 at 05:02 -0700, Gary Thomas wrote:
>>>> On 2014-11-27 01:35, Mike Looijmans wrote:
>>>>> Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
>>>>> deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
>>>>> now I finally figured out what causes it.
>>>>>
>>>>> (cut here)
>>>>>
>>>>> SUMMARY = "Demonstrate a bug in OE deployment"
>>>>> DESCRIPTION = "Build this package for a machine X, then look at the image's \
>>>>>     deploy directory. You'll see a deployme.txt there. Now build it for another \
>>>>>     machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
>>>>>     from the image dir. This appears to be a bug in OE's deployment."
>>>>> LICENSE = "GPLv2"
>>>>> LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
>>>>>
>>>>> inherit allarch deploy
>>>>>
>>>>> do_compile () {
>>>>>        echo "Hello world!" > deployme.txt
>>>>> }
>>>>>
>>>>> do_deploy () {
>>>>>        install -d ${DEPLOYDIR}
>>>>>        install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
>>>>> }
>>>>>
>>>>> addtask deploy before do_build after do_compile
>>>>>
>>>>> (cut here)
>>>>
>>>> Very interesting & verified with the latest master.
>>>>
>>>> Have you filed a bug?  https://bugzilla.yoctoproject.org/
>>>
>>> Well, I'm not convinced this is a bug as such. You've created an
>>> "allarch" deploy task, how would you expect this to behave?
>>>
>>> "allarch" means that the output from this task is universal and can be
>>> used on all targets. It will therefore get run once.
>>>
>>> A "deploy" task is machine specific.
>>>
>>> What ends up happening is therefore the task has a stamp is
>>> "universally" created. When you change machine, the checksum of the task
>>> changes, the previous version is removed, the new version is installed.
>>>
>>> So in many ways the system is doing exactly what I would expect it to do
>>> and it isn't a bug in that sense.
>>
>> It's not a bug in the sense that it doesn't do as it was programmed to do.
>
> Its doing *exactly* what the was designed to do. That doesn't match what
> you want/expect though.
>
>>   I understand what's happening here.
>>
>> I just think that the logic here is wrong. If "deploy" is machine specific,
>> then the implicit "undeploy" should be machine specific too, right?
>
> Well, its more complex than that.
>
> deploy.bbclass defaults to DEPLOY_DIR_IMAGE.
>
> DEPLOY_DIR_IMAGE defaults to ${DEPLOY_DIR}/images/${MACHINE}
>
> I actually put off merging the latter since I knew it would cause
> issues, I just couldn't articulate all of them at the time :(.
>
>>> The real question is how should an "allarch" + "deploy" task behave when
>>> you've specified machine specific paths? Perhaps erroring would be
>>> better?
>>
>> That would mean that roughly all deploy tasks will fail.
>
> I'm not sure we have many deploy+allarch tasks so I think "roughly none"
> would be a better description.
>
> deploy is usually used for bootloaders and kernels, both of which are
> not allarch.
>
>> At best they're tied
>> to MACHINE_ARCH, but never to MACHINE itself.
>
> No, they're tried to MACHINE itself, see above.
>
>> Would be strange to put PACKAGE_ARCH="${MACHINE}" in a recipe that clearly has
>> no dependency on machine specific things. And I wrote "${MACHINE}" here on
>> purpose.
>
> Dropping the "allarch" would be better than that.
>
>> I was thinking along the lines of "DEPLOY_DIR_IMAGE must have the same prefix"
>> or so.
>>
>> If I knew the solution, I'd have posted a patch instead of a question or report.
>
> Well, allarch.bbclass could override DEPLOY_DIR_IMAGE to remove
> the /${MACHINE} suffix. That would appear to fix the issues you're
> seeing, at the risk of having a different group of people upset that you
> don't get a complete directory per machine.
>
> It comes down to which behaviour we want. Changing MACHINE in the
> definition of DEPLOY_DIR_IMAGE to PACKAGE_ARCH might be the better
> solution, then it will deploy based on how specific or not specific the
> resulting code it. That will likely upset certain people even more
> though. The other option is to accept that its machine specific and do
> PACKAGE_ARCH="${MACHINE_ARCH}" in the class. That is suboptimal for the
> reasons you describe but would get the behaviour some people want.
>
> I suspect there isn't a right answer here :(
>
> An opinion from Martin/Koen might be useful at this point since it could
> affect them more than others (as well as things like the yocto project
> autobuilder output and output processing/testing/releasing).


If I understand correctly, having two "MACHINE" share the same 
MACHINE_ARCH (which is the case for several intel boards too), is not 
actually allowed?

Because you get the same kernel/bootloader for different machines then. 
That's what I wanted, and that triggered me finding this issue.

Please realize that the example is just an example. The actual problem 
I'm experiencing is with the kernel and bootloader!


-- 
Mike Looijmans

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Richard Purdie]

On Thu, 2014-11-27 at 16:17 +0100, Mike Looijmans wrote:
> If I understand correctly, having two "MACHINE" share the same 
> MACHINE_ARCH (which is the case for several intel boards too), is not 
> actually allowed?
> 
> Because you get the same kernel/bootloader for different machines then. 
> That's what I wanted, and that triggered me finding this issue.

Its a very good question. As things stand today, there is a one to one
mapping between MACHINE_ARCH and MACHINE, the difference being the
removal of invalid characters for the package namespace. That implies
that no, its not allowed.

The intel boards you mention do something a little different, they
inject a new "arch" into the hierarchy and then several machines share
that common "arch".

I suspect the intel boards you mention do suffer from the issue you
mention in a related way and the fix may be to s/MACHINE/PACKAGE_ARCH/
in DEPLOY_DIR_IMAGE as previously mentioned. The difference is I suspect
they deploy for the first machine built and then not for any other.
Nitin/Darren may be interested in checking into that.

> Please realize that the example is just an example. The actual problem 
> I'm experiencing is with the kernel and bootloader!

Right, the allarch behaviour is an illustration but not the underlying
problem you need to resolve (although its related).

Cheers,

Richard

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

On 11/27/2014 05:21 PM, Richard Purdie wrote:
> On Thu, 2014-11-27 at 16:17 +0100, Mike Looijmans wrote:
>> If I understand correctly, having two "MACHINE" share the same
>> MACHINE_ARCH (which is the case for several intel boards too), is not
>> actually allowed?
>>
>> Because you get the same kernel/bootloader for different machines then.
>> That's what I wanted, and that triggered me finding this issue.
>
> Its a very good question. As things stand today, there is a one to one
> mapping between MACHINE_ARCH and MACHINE, the difference being the
> removal of invalid characters for the package namespace. That implies
> that no, its not allowed.

Okay... So I have to revert that, even through it worked remarkably well so 
far (until I found this issue - compared to the situation before that, this is 
still a vast improvement).

> The intel boards you mention do something a little different, they
> inject a new "arch" into the hierarchy and then several machines share
> that common "arch".

Yeah. I did basically the opposite, I expanded the MACHINE with an extra 
suffix and then stripped the suffix away when composing the MACHINE_ARCH.

> I suspect the intel boards you mention do suffer from the issue you
> mention in a related way and the fix may be to s/MACHINE/PACKAGE_ARCH/
> in DEPLOY_DIR_IMAGE as previously mentioned. The difference is I suspect
> they deploy for the first machine built and then not for any other.
> Nitin/Darren may be interested in checking into that.

Yes.

But if I set DEPOY_DIR_IMAGE to use MACHINE_ARCH, I get the opposite problem 
of two MACHINEs overwriting eachothers root filesystem. Though the $MACHINE in 
most of the filenames (which is typically removed for the links) might help 
separate those.

>> Please realize that the example is just an example. The actual problem
>> I'm experiencing is with the kernel and bootloader!
>
> Right, the allarch behaviour is an illustration but not the underlying
> problem you need to resolve (although its related).

Well, I thought I had it all sorted out, but I'm back to square one now.

What I'm really trying to do here is to create another "dimension".

The board contains a combined ARM + FPGA. The ARM part is just like any other 
machine, and it's the same on all boards.

The FPGA part also needs things built in the OE environment, and it has its 
own OS and "applications" in the form of a static bitstream to configure the 
whole device, and partial bitstreams that are swapped in and out at runtime to 
use it for various different functions. The build structure works much like a 
normal CPU toolchain, in that it creates the full bitstream which provides the 
"library" (Xilinx calls this a checkpoint) for the partials.
The FPGA can be of different types (currently we have xc7z015 and xc7z030 
boards, but there are plans to create SOMs for various other types too), and 
the bistreams are unique for each type, unlike CPU code, you cannot exchange 
them. The FPGA recipes leave these checkpoints into the sysroot where other 
FPGA recipes can pick them up and expand on them. Hence, I can't just make 
separate recipes for different FPGA types, because they need the sysroot to be 
specific to them.

So what I did was define the MACHINE as a combination of SOM+carrier+FPGA. The 
SOM+carrier part went into SOM_FAMILY and MACHINE_ARCH, and the FPGA part into 
the "FPGA_FAMILY" (and an FPGA_FAMILY_ARCH). The FPGA recipes use PACKAGE_ARCH 
= "${FPGA_FAMILY_ARCH}".

This config file sets that up:
https://github.com/topic-embedded-products/meta-topic/blob/master/conf/machine/include/topic-miami.inc

This has so far accomplished what I wanted to do, It makes all the ARM part 
things (like kernel and bootloader) generic, because they don't depend on the 
type of FPGA at all. And it makes the FPGA parts independent of the ARM side 
too, drastically reducing build times (FPGA bitstreams typically take 1 to 4 
hours to build, and that's for the smaller devices..) and heavily leaning on 
shares sstate-cache so we can re-use a buildserver's nightly output all over 
the place.

Another approach would be to introduce a global variable (FPGA_FAMILY?) that 
has the same level as MACHINE does, and require it to be set in local.conf or 
environment.
By explicitly using FPGA_FAMILY in intermediate and output paths, I think I 
can make it so that the sysroot can still hold the intermediate files for 
various targets. And simply including FPGA_FAMILY in the rootfs image name 
should solve the deployment issue.

I think more machines will emerge that need this kind of setup. The kernel has 
already gotten quite good at supporting hardware that can just grow itself a 
network card or i2c controller when it needs one...


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Richard Purdie]

On Fri, 2014-11-28 at 08:09 +0100, Mike Looijmans wrote:
> On 11/27/2014 05:21 PM, Richard Purdie wrote:
> > I suspect the intel boards you mention do suffer from the issue you
> > mention in a related way and the fix may be to s/MACHINE/PACKAGE_ARCH/
> > in DEPLOY_DIR_IMAGE as previously mentioned. The difference is I suspect
> > they deploy for the first machine built and then not for any other.
> > Nitin/Darren may be interested in checking into that.
> 
> Yes.
> 
> But if I set DEPOY_DIR_IMAGE to use MACHINE_ARCH, I get the opposite problem 
> of two MACHINEs overwriting eachothers root filesystem. Though the $MACHINE in 
> most of the filenames (which is typically removed for the links) might help 
> separate those.

Ok, I hadn't realised the rootfs was different in these cases (although
it makes sense).

> >> Please realize that the example is just an example. The actual problem
> >> I'm experiencing is with the kernel and bootloader!
> >
> > Right, the allarch behaviour is an illustration but not the underlying
> > problem you need to resolve (although its related).
> 
> Well, I thought I had it all sorted out, but I'm back to square one now.
> 
> What I'm really trying to do here is to create another "dimension".
>
> The board contains a combined ARM + FPGA. The ARM part is just like any other 
> machine, and it's the same on all boards.
> 
> The FPGA part also needs things built in the OE environment, and it has its 
> own OS and "applications" in the form of a static bitstream to configure the 
> whole device, and partial bitstreams that are swapped in and out at runtime to 
> use it for various different functions. The build structure works much like a 
> normal CPU toolchain, in that it creates the full bitstream which provides the 
> "library" (Xilinx calls this a checkpoint) for the partials.
> The FPGA can be of different types (currently we have xc7z015 and xc7z030 
> boards, but there are plans to create SOMs for various other types too), and 
> the bistreams are unique for each type, unlike CPU code, you cannot exchange 
> them. The FPGA recipes leave these checkpoints into the sysroot where other 
> FPGA recipes can pick them up and expand on them. Hence, I can't just make 
> separate recipes for different FPGA types, because they need the sysroot to be 
> specific to them.
> 
> So what I did was define the MACHINE as a combination of SOM+carrier+FPGA. The 
> SOM+carrier part went into SOM_FAMILY and MACHINE_ARCH, and the FPGA part into 
> the "FPGA_FAMILY" (and an FPGA_FAMILY_ARCH). The FPGA recipes use PACKAGE_ARCH 
> = "${FPGA_FAMILY_ARCH}".
> 
> This config file sets that up:
> https://github.com/topic-embedded-products/meta-topic/blob/master/conf/machine/include/topic-miami.inc
> 
> This has so far accomplished what I wanted to do, It makes all the ARM part 
> things (like kernel and bootloader) generic, because they don't depend on the 
> type of FPGA at all. And it makes the FPGA parts independent of the ARM side 
> too, drastically reducing build times (FPGA bitstreams typically take 1 to 4 
> hours to build, and that's for the smaller devices..) and heavily leaning on 
> shares sstate-cache so we can re-use a buildserver's nightly output all over 
> the place.
> 
> Another approach would be to introduce a global variable (FPGA_FAMILY?) that 
> has the same level as MACHINE does, and require it to be set in local.conf or 
> environment.
> By explicitly using FPGA_FAMILY in intermediate and output paths, I think I 
> can make it so that the sysroot can still hold the intermediate files for 
> various targets. And simply including FPGA_FAMILY in the rootfs image name 
> should solve the deployment issue.
> 
> I think more machines will emerge that need this kind of setup. The kernel has 
> already gotten quite good at supporting hardware that can just grow itself a 
> network card or i2c controller when it needs one...

At the risk of repeating myself, I still think what may work best is
making the kernel and bootloader specific to their own specific arch,
like meta-intel does. That way the machines will share the kernel and
bootloader but still have the correct namespaces. You can do something
similar for the FPGAs so they also have their own namespaces too.

Changing the top level control (MACHINE), whilst initially appealing is
problematic and whilst I can see more machines appearing like this, I
cannot see us wanting to add a new top level control to cater for it as
it will cause all kinds of problems. Even the naming is fraught with
problems (FPGA is too specific, it may not be a SOC, etc). I went
through this discussion with the Intel people and this is why meta-intel
inserted the level it did. This wasn't their first choice either, but
the alternatives don't make sense when you try and code them (and
consider the implications for usage of the system, documentation and so
on).

On the other hand, inserting a new level under MACHINE is well supported
and comparatively trivial. If this has issues, I believe we can fix them
without disrupting as much of the system, we just need to understand
what they are.

So I guess I'd ask why the level under MACHINE doesn't work for you and
what we can do to improve it?

Cheers,

Richard

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

On 11/28/2014 11:18 AM, Richard Purdie wrote:
> On Fri, 2014-11-28 at 08:09 +0100, Mike Looijmans wrote:
...
>> What I'm really trying to do here is to create another "dimension".
>>
>> The board contains a combined ARM + FPGA. The ARM part is just like any other
>> machine, and it's the same on all boards.
>>
>> The FPGA part also needs things built in the OE environment, and it has its
>> own OS and "applications" in the form of a static bitstream to configure the
>> whole device, and partial bitstreams that are swapped in and out at runtime to
>> use it for various different functions. The build structure works much like a
>> normal CPU toolchain, in that it creates the full bitstream which provides the
>> "library" (Xilinx calls this a checkpoint) for the partials.
>> The FPGA can be of different types (currently we have xc7z015 and xc7z030
>> boards, but there are plans to create SOMs for various other types too), and
>> the bistreams are unique for each type, unlike CPU code, you cannot exchange
>> them. The FPGA recipes leave these checkpoints into the sysroot where other
>> FPGA recipes can pick them up and expand on them. Hence, I can't just make
>> separate recipes for different FPGA types, because they need the sysroot to be
>> specific to them.
>>
>> So what I did was define the MACHINE as a combination of SOM+carrier+FPGA. The
>> SOM+carrier part went into SOM_FAMILY and MACHINE_ARCH, and the FPGA part into
>> the "FPGA_FAMILY" (and an FPGA_FAMILY_ARCH). The FPGA recipes use PACKAGE_ARCH
>> = "${FPGA_FAMILY_ARCH}".
>>
>> This config file sets that up:
>> https://github.com/topic-embedded-products/meta-topic/blob/master/conf/machine/include/topic-miami.inc
>>
>> This has so far accomplished what I wanted to do, It makes all the ARM part
>> things (like kernel and bootloader) generic, because they don't depend on the
>> type of FPGA at all. And it makes the FPGA parts independent of the ARM side
>> too, drastically reducing build times (FPGA bitstreams typically take 1 to 4
>> hours to build, and that's for the smaller devices..) and heavily leaning on
>> shares sstate-cache so we can re-use a buildserver's nightly output all over
>> the place.
>>
>> Another approach would be to introduce a global variable (FPGA_FAMILY?) that
>> has the same level as MACHINE does, and require it to be set in local.conf or
>> environment.
>> By explicitly using FPGA_FAMILY in intermediate and output paths, I think I
>> can make it so that the sysroot can still hold the intermediate files for
>> various targets. And simply including FPGA_FAMILY in the rootfs image name
>> should solve the deployment issue.
>>
>> I think more machines will emerge that need this kind of setup. The kernel has
>> already gotten quite good at supporting hardware that can just grow itself a
>> network card or i2c controller when it needs one...
>
> At the risk of repeating myself, I still think what may work best is
> making the kernel and bootloader specific to their own specific arch,
> like meta-intel does. That way the machines will share the kernel and
> bootloader but still have the correct namespaces. You can do something
> similar for the FPGAs so they also have their own namespaces too.
>
> Changing the top level control (MACHINE), whilst initially appealing is
> problematic and whilst I can see more machines appearing like this, I
> cannot see us wanting to add a new top level control to cater for it as
> it will cause all kinds of problems. Even the naming is fraught with
> problems (FPGA is too specific, it may not be a SOC, etc). I went

Agree FPGA is a bad word in this context, there could be multiple of 
these devices, and they could be DSPs or even CPUs. I'm going to keep 
calling it FPGA here, just for lack of a generic term like "secundary 
system" or so (tertiary?).

> through this discussion with the Intel people and this is why meta-intel
> inserted the level it did. This wasn't their first choice either, but
> the alternatives don't make sense when you try and code them (and
> consider the implications for usage of the system, documentation and so
> on).
>
> On the other hand, inserting a new level under MACHINE is well supported
> and comparatively trivial. If this has issues, I believe we can fix them
> without disrupting as much of the system, we just need to understand
> what they are.
>
> So I guess I'd ask why the level under MACHINE doesn't work for you and
> what we can do to improve it?

I tried going that way. I came to the conclusion that I wanted EVERY 
package that said it was MACHINE_ARCH was to be moved to the lower 
level. So that would be the same as simply setting, for a MACHINE="x-1"
MACHINE_ARCH="x". That accomplished what I wanted - the "1" part no 
longer mattered for any package that did not know about the FPGA part 
explicitly. The packages for the FPGA end up with PACKAGE_ARCH="1" in 
this case, so they're compatible with any other machine that carries the 
same FPGA part.

What I did was basically the same as the intel layer, only the intel 
layer only lowered the MACHINE_ARCH for bootloader and kernel, while I 
just set the whitelist of package to move to "all".

We ARE quite used to having these sub-systems around. Think about the 
firmware for GPU and WIFI chips for example. These are generally just 
binary blobs that magically appear out of the hat of a vendor. Now 
imagine that these were open source too, and that you have the option to 
build the firmware for the wifi or GPU chip from source in the build 
environment. How would you do that? Maybe a "sub" openembedded tree that 
builds the crosscompiler for it? Could I call bitbake recursively?
That's basically the same thing that's going on here.

Currently I'm trying to do the FPGA builds as the OMAPs do for the DSP, 
to download the tools and build the DSP code using these tools installed 
in the sysroot for the machine. But these packages just mention they're 
machine specific, which is not quite correct, since they're intended for 
the DSP on it.

These subsystems are totally unrelated to the main cpu system, and are 
orthogonal. You could have a system with a cpu, gpu, wifi and fpga, and 
that'd make 4 dimensions. The machines with the same GPU can share the 
binary result of that, the ones with the same wifi chip can share that 
(if they have a compatible driver...), etcetera.

Reading back my post now, it looks as though "dimensions" are what I 
really need to make this work as I'd like. Something like "MACHINE" 
being an array of sub-components:

MACHINE=["topic-miami", "xc7z015", ...]

That would also open up a path to have click-on boards (like many EVM 
boards support, e.g. pmods and FMC boards) in that list. These addon 
boards can usually be added to various machines, and aren't limited to 
certain architectures, but the functionality they provide is the same, 
and also the tools and drivers they'll need to function properly.

-- 
Mike Looijmans

Re: Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

On 11/27/2014 03:41 PM, Richard Purdie wrote:
> On Thu, 2014-11-27 at 15:22 +0100, Mike Looijmans wrote:
>> On 11/27/2014 02:17 PM, Richard Purdie wrote:
>>> On Thu, 2014-11-27 at 05:02 -0700, Gary Thomas wrote:
>>>> On 2014-11-27 01:35, Mike Looijmans wrote:
>>>>> Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
>>>>> deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
>>>>> now I finally figured out what causes it.
>>>>>
>>>>> (cut here)
>>>>>
>>>>> SUMMARY = "Demonstrate a bug in OE deployment"
>>>>> DESCRIPTION = "Build this package for a machine X, then look at the image's \
>>>>>     deploy directory. You'll see a deployme.txt there. Now build it for another \
>>>>>     machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
>>>>>     from the image dir. This appears to be a bug in OE's deployment."
>>>>> LICENSE = "GPLv2"
>>>>> LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
>>>>>
>>>>> inherit allarch deploy
>>>>>
>>>>> do_compile () {
>>>>>        echo "Hello world!" > deployme.txt
>>>>> }
>>>>>
>>>>> do_deploy () {
>>>>>        install -d ${DEPLOYDIR}
>>>>>        install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
>>>>> }
>>>>>
>>>>> addtask deploy before do_build after do_compile
>>>>>
>>>>> (cut here)
>>>>
>>>> Very interesting & verified with the latest master.
>>>>
>>>> Have you filed a bug?  https://bugzilla.yoctoproject.org/
>>>
>>> Well, I'm not convinced this is a bug as such. You've created an
>>> "allarch" deploy task, how would you expect this to behave?
>>>
>>> "allarch" means that the output from this task is universal and can be
>>> used on all targets. It will therefore get run once.
>>>
>>> A "deploy" task is machine specific.
>>>
>>> What ends up happening is therefore the task has a stamp is
>>> "universally" created. When you change machine, the checksum of the task
>>> changes, the previous version is removed, the new version is installed.
>>>
>>> So in many ways the system is doing exactly what I would expect it to do
>>> and it isn't a bug in that sense.
>>
>> It's not a bug in the sense that it doesn't do as it was programmed to do.
>
> Its doing *exactly* what the was designed to do. That doesn't match what
> you want/expect though.
>
>>   I understand what's happening here.
>>
>> I just think that the logic here is wrong. If "deploy" is machine specific,
>> then the implicit "undeploy" should be machine specific too, right?
>
> Well, its more complex than that.
>
> deploy.bbclass defaults to DEPLOY_DIR_IMAGE.
>
> DEPLOY_DIR_IMAGE defaults to ${DEPLOY_DIR}/images/${MACHINE}
>
> I actually put off merging the latter since I knew it would cause
> issues, I just couldn't articulate all of them at the time :(.
>
>>> The real question is how should an "allarch" + "deploy" task behave when
>>> you've specified machine specific paths? Perhaps erroring would be
>>> better?
>>
>> That would mean that roughly all deploy tasks will fail.
>
> I'm not sure we have many deploy+allarch tasks so I think "roughly none"
> would be a better description.
>
> deploy is usually used for bootloaders and kernels, both of which are
> not allarch.
>
>> At best they're tied
>> to MACHINE_ARCH, but never to MACHINE itself.
>
> No, they're tried to MACHINE itself, see above.
>
>> Would be strange to put PACKAGE_ARCH="${MACHINE}" in a recipe that clearly has
>> no dependency on machine specific things. And I wrote "${MACHINE}" here on
>> purpose.
>
> Dropping the "allarch" would be better than that.
>
>> I was thinking along the lines of "DEPLOY_DIR_IMAGE must have the same prefix"
>> or so.
>>
>> If I knew the solution, I'd have posted a patch instead of a question or report.
>
> Well, allarch.bbclass could override DEPLOY_DIR_IMAGE to remove
> the /${MACHINE} suffix. That would appear to fix the issues you're
> seeing, at the risk of having a different group of people upset that you
> don't get a complete directory per machine.
>
> It comes down to which behaviour we want. Changing MACHINE in the
> definition of DEPLOY_DIR_IMAGE to PACKAGE_ARCH might be the better
> solution, then it will deploy based on how specific or not specific the
> resulting code it. That will likely upset certain people even more
> though. The other option is to accept that its machine specific and do
> PACKAGE_ARCH="${MACHINE_ARCH}" in the class. That is suboptimal for the
> reasons you describe but would get the behaviour some people want.

We might be able to reduce the number of upset people drastically by doing the 
deployment in two stages.

Step one is to deploy the binaries to ${DEPLOY_DIR}/${PACKAGE_ARCH} as per 
your suggestion. Having pondered about this for many days now, that is 
actually the only thing that is "right" from a technical point of view.

That will wreak havoc on all scripts that want to take the deployment results 
and copy them to target devices (I have quite a few of those scripts myself), 
because these now suddenly need to become aware of which package is for what 
architecture.

That can be solved in a second step: For the MACHINE, create symbolic links 
from ${DEPLOY_DIR}/${MACHINE} to the files in ${DEPLOY_DIR}/${PACKAGE_ARCH}. 
Most packages (at least, bootloader and kernel do), already create symlinks to 
actual targets, so I'd expect existing scripts to be able to cope with that 
already. This second step is purely MACHINE specific, and only serves to help 
gather everything that is needed for a machine deployment. This second step is 
to be rerun for each MACHINE.

This second step could even be automated, one could iterate through all 
supported ARCH for the machine, and create a symlink for each and every file 
found there in the ${DEPLOY_DIR}/${MACHINE} directory.


The only problem I can see with this approach is that you may end up with 
dangling symlinks for machine B after updating and deploying a shared package 
for machine A, or worse, a version mismatch of the deployed target.



> An opinion from Martin/Koen might be useful at this point since it could
> affect them more than others (as well as things like the yocto project
> autobuilder output and output processing/testing/releasing).

So far, we haven't heard from them :(



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/