[devtool][PATCH] devtool deploy-target: optionally specify package

[devtool][PATCH] devtool deploy-target: optionally specify package

[Trevor Woerner]

Instead of installing an entire recipe's build output (i.e. ${D}), allow the
user to optionally specify a package from said recipe to be installed
exclusively (i.e. ${PKGDEST}/<package>).

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 scripts/lib/devtool/deploy.py | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
index 52e261d560..c022757519 100644
--- a/scripts/lib/devtool/deploy.py
+++ b/scripts/lib/devtool/deploy.py
@@ -169,11 +169,20 @@ def deploy(args, config, basepath, workspace):
         except Exception as e:
             raise DevtoolError('Exception parsing recipe %s: %s' %
                             (args.recipename, e))
-        recipe_outdir = rd.getVar('D')
+        if args.package:
+            recipe_outdir = os.path.join(rd.getVar('PKGDEST'), args.package)
+        else:
+            recipe_outdir = rd.getVar('D')
         if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir):
-            raise DevtoolError('No files to deploy - have you built the %s '
-                            'recipe? If so, the install step has not installed '
-                            'any files.' % args.recipename)
+            if args.package:
+                raise DevtoolError('No files to deploy - have you built the %s '
+                                'package of the %s recipe? If so, the install '
+                                'step has not installed any files.'
+                                % (args.package, args.recipename))
+            else:
+                raise DevtoolError('No files to deploy - have you built the %s '
+                                'recipe? If so, the install step has not installed '
+                                'any files.' % args.recipename)
 
         if args.strip and not args.dry_run:
             # Fakeroot copy to new destination
@@ -314,8 +323,8 @@ def register_commands(subparsers, context):
     """Register devtool subcommands from the deploy plugin"""
 
     parser_deploy = subparsers.add_parser('deploy-target',
-                                          help='Deploy recipe output files to live target machine',
-                                          description='Deploys a recipe\'s build output (i.e. the output of the do_install task) to a live target machine over ssh. By default, any existing files will be preserved instead of being overwritten and will be restored if you run devtool undeploy-target. Note: this only deploys the recipe itself and not any runtime dependencies, so it is assumed that those have been installed on the target beforehand.',
+                                          help='Deploy build output to a live target machine',
+                                          description='Deploys either the full recipe\'s build output (i.e. the output of the do_install task) or a package of a recipe to a live target machine over ssh. By default, any existing files will be preserved instead of being overwritten and will be restored if you run devtool undeploy-target. Note: this only deploys the specified item itself and not any runtime dependencies, so it is assumed that those have been installed on the target beforehand.',
                                           group='testbuild')
     parser_deploy.add_argument('recipename', help='Recipe to deploy')
     parser_deploy.add_argument('target', help='Live target machine running an ssh server: user@hostname[:destdir]')
@@ -325,6 +334,7 @@ def register_commands(subparsers, context):
     parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
     parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true')
     parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
+    parser_deploy.add_argument('--package', help='Specify a recipe\'s package to deploy', dest='package')
 
     strip_opts = parser_deploy.add_mutually_exclusive_group(required=False)
     strip_opts.add_argument('-S', '--strip',
@@ -337,8 +347,8 @@ def register_commands(subparsers, context):
     parser_deploy.set_defaults(func=deploy)
 
     parser_undeploy = subparsers.add_parser('undeploy-target',
-                                            help='Undeploy recipe output files in live target machine',
-                                            description='Un-deploys recipe output files previously deployed to a live target machine by devtool deploy-target.',
+                                            help='Undeploy output files from a live target machine',
+                                            description='Un-deploys output files previously deployed to a live target machine by devtool deploy-target.',
                                             group='testbuild')
     parser_undeploy.add_argument('recipename', help='Recipe to undeploy (if not using -a/--all)', nargs='?')
     parser_undeploy.add_argument('target', help='Live target machine running an ssh server: user@hostname')
-- 
2.17.0.582.gccdcbd54c

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Andre McCurdy]

On Thu, May 31, 2018 at 1:06 PM, Trevor Woerner <twoerner@gmail.com> wrote:
> Instead of installing an entire recipe's build output (i.e. ${D}), allow the
> user to optionally specify a package from said recipe to be installed
> exclusively (i.e. ${PKGDEST}/<package>).

I'm not a big devtool user myself, but I'm curious about the
situations where you might need this?

Are the going to be problems if users want to deploy more than one
package (or switch between a full deploy and a package) and don't
deploy and undeploy in the right order?

> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---
>  scripts/lib/devtool/deploy.py | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
>
> diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
> index 52e261d560..c022757519 100644
> --- a/scripts/lib/devtool/deploy.py
> +++ b/scripts/lib/devtool/deploy.py
> @@ -169,11 +169,20 @@ def deploy(args, config, basepath, workspace):
>          except Exception as e:
>              raise DevtoolError('Exception parsing recipe %s: %s' %
>                              (args.recipename, e))
> -        recipe_outdir = rd.getVar('D')
> +        if args.package:
> +            recipe_outdir = os.path.join(rd.getVar('PKGDEST'), args.package)
> +        else:
> +            recipe_outdir = rd.getVar('D')
>          if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir):
> -            raise DevtoolError('No files to deploy - have you built the %s '
> -                            'recipe? If so, the install step has not installed '
> -                            'any files.' % args.recipename)
> +            if args.package:
> +                raise DevtoolError('No files to deploy - have you built the %s '
> +                                'package of the %s recipe? If so, the install '
> +                                'step has not installed any files.'
> +                                % (args.package, args.recipename))
> +            else:
> +                raise DevtoolError('No files to deploy - have you built the %s '
> +                                'recipe? If so, the install step has not installed '
> +                                'any files.' % args.recipename)
>
>          if args.strip and not args.dry_run:
>              # Fakeroot copy to new destination
> @@ -314,8 +323,8 @@ def register_commands(subparsers, context):
>      """Register devtool subcommands from the deploy plugin"""
>
>      parser_deploy = subparsers.add_parser('deploy-target',
> -                                          help='Deploy recipe output files to live target machine',
> -                                          description='Deploys a recipe\'s build output (i.e. the output of the do_install task) to a live target machine over ssh. By default, any existing files will be preserved instead of being overwritten and will be restored if you run devtool undeploy-target. Note: this only deploys the recipe itself and not any runtime dependencies, so it is assumed that those have been installed on the target beforehand.',
> +                                          help='Deploy build output to a live target machine',
> +                                          description='Deploys either the full recipe\'s build output (i.e. the output of the do_install task) or a package of a recipe to a live target machine over ssh. By default, any existing files will be preserved instead of being overwritten and will be restored if you run devtool undeploy-target. Note: this only deploys the specified item itself and not any runtime dependencies, so it is assumed that those have been installed on the target beforehand.',
>                                            group='testbuild')
>      parser_deploy.add_argument('recipename', help='Recipe to deploy')
>      parser_deploy.add_argument('target', help='Live target machine running an ssh server: user@hostname[:destdir]')
> @@ -325,6 +334,7 @@ def register_commands(subparsers, context):
>      parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
>      parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true')
>      parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
> +    parser_deploy.add_argument('--package', help='Specify a recipe\'s package to deploy', dest='package')
>
>      strip_opts = parser_deploy.add_mutually_exclusive_group(required=False)
>      strip_opts.add_argument('-S', '--strip',
> @@ -337,8 +347,8 @@ def register_commands(subparsers, context):
>      parser_deploy.set_defaults(func=deploy)
>
>      parser_undeploy = subparsers.add_parser('undeploy-target',
> -                                            help='Undeploy recipe output files in live target machine',
> -                                            description='Un-deploys recipe output files previously deployed to a live target machine by devtool deploy-target.',
> +                                            help='Undeploy output files from a live target machine',
> +                                            description='Un-deploys output files previously deployed to a live target machine by devtool deploy-target.',
>                                              group='testbuild')
>      parser_undeploy.add_argument('recipename', help='Recipe to undeploy (if not using -a/--all)', nargs='?')
>      parser_undeploy.add_argument('target', help='Live target machine running an ssh server: user@hostname')
> --
> 2.17.0.582.gccdcbd54c
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Trevor Woerner]

[-- Attachment #1: Type: text/plain, Size: 4331 bytes --]

Hi Andre, thanks for taking an interest!

On Thu, May 31, 2018 at 6:22 PM, Andre McCurdy <armccurdy@gmail.com> wrote:

> On Thu, May 31, 2018 at 1:06 PM, Trevor Woerner <twoerner@gmail.com>
> wrote:
> > Instead of installing an entire recipe's build output (i.e. ${D}), allow
> the
> > user to optionally specify a package from said recipe to be installed
> > exclusively (i.e. ${PKGDEST}/<package>).
>
> I'm not a big devtool user myself, but I'm curious about the
> situations where you might need this?
>

Earlier today I was working on recipes for some software which recently
added the feature to dlopen() one of a couple shared objects based on
various conditions. But instead of dlopen()'ing the SONAME, they were
dlopen()'ing the raw .so file. OE's default packaging, of course, added the
*.so.0 and *.so.0.0.0 files to the ${PN} package, and the raw *.so files to
the ${PN}-dev package. So when the app tried to run, it failed.

Investigating, I realised there were two things I could do: 1) add the raw
*.so files to the ${PN} package (with INSANE_SKIP += "dev-so") or 2) fix
the app. I decided to try #2, with the help of devtool.

I made one small change to the app, "devtool deploy-target"'ed it, and,
AMAZINGLY, everything worked!! Wow, what luck! ... or not.

Investigating, I discovered that "devtool deploy-target"'s only mode of
operation is to upload the entire recipe's ${D} directory (i.e.
${WORKDIR}/image) to the target! I hope you'll be as surprised by that as
am I. Considering ${D} includes *.a files, man pages, raw *.so files,
pkgconfigs, (... thankfully it doesn't also include the -dbg packages) that
is a lot of stuff to be uploading to the target, a lot of wrong stuff, and
in my case that hindered my work. Most target filesystems are only 10%
larger than required to hold their contents, so uploading static libraries,
man pages, pkgconfigs, etc is not great.

Specifically, these unwanted raw *.so files were hindering my work. My only
two options were: 1) to use "devtool deploy-target" as-is and then have to
manually delete all the .so files or 2) come up with a better solution. I
went with #2 (again).

Many recipes generate just a couple packages: ${PN}, -dbg, -doc, -dev,
-staticdev, -locale. But some generate a whole bunch. In my case my recipe
is generating a lot of packages, but I only wanted to investigate one. This
patch allows a developer to focus on just one package, instead of the
entire recipe.


> Are the going to be problems if users want to deploy more than one
> package (or switch between a full deploy and a package) and don't
> deploy and undeploy in the right order?
>

In my patch I added a --package option to "devtool deploy-target" but not
to "devtool undeploy-target". devtool keeps track of what files were
uploaded by creating a per-package list on the target itself (in
/.devtool). When you undeploy a recipe, it simply looks for this recipe
file on the target, reads the files listed there, and undeploys each of the
files in that list.

Therefore, mixing and matching deploys and undeploys amongst recipes don't
interfere with each other.

I checked at the time, but I'm not 100% sure now, but I believe if one
deploys two packages from one recipe, the on-target file gets appended. But
now I'm not 100% sure, so I'll verify this tomorrow (thanks for the
reminder). As such, under my scheme, you can deploy recipes and packages at
will, but then only undeploy the whole kit and caboodle in one go; given a
recipe name to undeploy, whatever it finds in the on-target record will get
undeployed.

I think any developer will find this reasonable behaviour (but maybe not?).

In any case, my patch doesn't affect any existing workflows. If a developer
never notices the new --package option and simply continues to use "devtool
deploy-target" with just a recipe name, the behaviour will be exactly the
same. Only when the --package option is used will the new behaviour occur.

I'll verify deploying multiple packages from the same recipe to confirm
they append the list file. I'll also verify deploying the same package
multiple times (although my guess is there'll be no difference between this
and the existing behaviour should one deploy the same recipe multiple
times).

[-- Attachment #2: Type: text/html, Size: 5524 bytes --]

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Andre McCurdy]

On Thu, May 31, 2018 at 7:23 PM, Trevor Woerner <twoerner@gmail.com> wrote:
> Hi Andre, thanks for taking an interest!
>
> On Thu, May 31, 2018 at 6:22 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
>>
>> On Thu, May 31, 2018 at 1:06 PM, Trevor Woerner <twoerner@gmail.com>
>> wrote:
>> > Instead of installing an entire recipe's build output (i.e. ${D}), allow
>> > the
>> > user to optionally specify a package from said recipe to be installed
>> > exclusively (i.e. ${PKGDEST}/<package>).
>>
>> I'm not a big devtool user myself, but I'm curious about the
>> situations where you might need this?
>
> Earlier today I was working on recipes for some software which recently
> added the feature to dlopen() one of a couple shared objects based on
> various conditions. But instead of dlopen()'ing the SONAME, they were
> dlopen()'ing the raw .so file. OE's default packaging, of course, added the
> *.so.0 and *.so.0.0.0 files to the ${PN} package, and the raw *.so files to
> the ${PN}-dev package. So when the app tried to run, it failed.

Sounds familiar. I used to maintain this patch for the mesa recipe to
workaround a similar issue in Qt:

  http://lists.openembedded.org/pipermail/openembedded-devel/2015-May/101455.html

I don't know if later versions of Qt still show the problem (and of
course, the workaround in the mesa recipe was never merged...).

> Investigating, I realised there were two things I could do: 1) add the raw
> *.so files to the ${PN} package (with INSANE_SKIP += "dev-so") or 2) fix the
> app. I decided to try #2, with the help of devtool.
>
> I made one small change to the app, "devtool deploy-target"'ed it, and,
> AMAZINGLY, everything worked!! Wow, what luck! ... or not.
>
> Investigating, I discovered that "devtool deploy-target"'s only mode of
> operation is to upload the entire recipe's ${D} directory (i.e.
> ${WORKDIR}/image) to the target! I hope you'll be as surprised by that as am
> I. Considering ${D} includes *.a files, man pages, raw *.so files,
> pkgconfigs, (... thankfully it doesn't also include the -dbg packages) that
> is a lot of stuff to be uploading to the target, a lot of wrong stuff, and
> in my case that hindered my work. Most target filesystems are only 10%
> larger than required to hold their contents, so uploading static libraries,
> man pages, pkgconfigs, etc is not great.
>
> Specifically, these unwanted raw *.so files were hindering my work. My only
> two options were: 1) to use "devtool deploy-target" as-is and then have to
> manually delete all the .so files or 2) come up with a better solution. I
> went with #2 (again).
>
> Many recipes generate just a couple packages: ${PN}, -dbg, -doc, -dev,
> -staticdev, -locale. But some generate a whole bunch. In my case my recipe
> is generating a lot of packages, but I only wanted to investigate one. This
> patch allows a developer to focus on just one package, instead of the entire
> recipe.

All seems reasonable to me. Thanks for taking the time to explain all
the details!

I wonder whether the default behaviour should just be to not deploy
files from the -dev package?

>> Are the going to be problems if users want to deploy more than one
>> package (or switch between a full deploy and a package) and don't
>> deploy and undeploy in the right order?
>
>
> In my patch I added a --package option to "devtool deploy-target" but not to
> "devtool undeploy-target". devtool keeps track of what files were uploaded
> by creating a per-package list on the target itself (in /.devtool). When you
> undeploy a recipe, it simply looks for this recipe file on the target, reads
> the files listed there, and undeploys each of the files in that list.
>
> Therefore, mixing and matching deploys and undeploys amongst recipes don't
> interfere with each other.
>
> I checked at the time, but I'm not 100% sure now, but I believe if one
> deploys two packages from one recipe, the on-target file gets appended. But
> now I'm not 100% sure, so I'll verify this tomorrow (thanks for the
> reminder). As such, under my scheme, you can deploy recipes and packages at
> will, but then only undeploy the whole kit and caboodle in one go; given a
> recipe name to undeploy, whatever it finds in the on-target record will get
> undeployed.
>
> I think any developer will find this reasonable behaviour (but maybe not?).

If that's how it works then it seems reasonable to me.

> In any case, my patch doesn't affect any existing workflows. If a developer
> never notices the new --package option and simply continues to use "devtool
> deploy-target" with just a recipe name, the behaviour will be exactly the
> same. Only when the --package option is used will the new behaviour occur.
>
> I'll verify deploying multiple packages from the same recipe to confirm they
> append the list file. I'll also verify deploying the same package multiple
> times (although my guess is there'll be no difference between this and the
> existing behaviour should one deploy the same recipe multiple times).

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Trevor Woerner]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

On Thu, May 31, 2018 at 11:07 PM, Andre McCurdy <armccurdy@gmail.com> wrote:

> On Thu, May 31, 2018 at 7:23 PM, Trevor Woerner <twoerner@gmail.com>
> wrote:
>
> I wonder whether the default behaviour should just be to not deploy
> files from the -dev package?
>
>
That would certainly be an improvement. But it still wouldn't make sense to
copy over the -staticdev or -doc packages either. Maybe a better default
behaviour would be to not copy over the -dev, -staticdev, -doc, -dbg, and
-locale packages? Or better yet: just copy over the core packages, not any
of the core-<something> packages? I'm guessing bitbake has a list somewhere
of these -somethings?

I wouldn't be surprised if the current behaviour exists simply because ${D}
is already there and ready to go. My tweak extends the behaviour to send
another already-there directory: ${PKGDEST} + <package>.

If we were to go this route (i.e. being more selective about which packages
are sent) do you think this should be the new default behaviour, or should
the existing default behaviour be preserved and a flag used for this tweak?




> >> Are the going to be problems if users want to deploy more than one
> >> package (or switch between a full deploy and a package) and don't
> >> deploy and undeploy in the right order?
> >
> >
> > In my patch I added a --package option to "devtool deploy-target" but
> not to
> > "devtool undeploy-target". devtool keeps track of what files were
> uploaded
> > by creating a per-package list on the target itself (in /.devtool). When
> you
> > undeploy a recipe, it simply looks for this recipe file on the target,
> reads
> > the files listed there, and undeploys each of the files in that list.
> >
> > Therefore, mixing and matching deploys and undeploys amongst recipes
> don't
> > interfere with each other.
> >
> > I checked at the time, but I'm not 100% sure now, but I believe if one
> > deploys two packages from one recipe, the on-target file gets appended.
> But
> > now I'm not 100% sure, so I'll verify this tomorrow (thanks for the
> > reminder). As such, under my scheme, you can deploy recipes and packages
> at
> > will, but then only undeploy the whole kit and caboodle in one go; given
> a
> > recipe name to undeploy, whatever it finds in the on-target record will
> get
> > undeployed.
> >
> > I think any developer will find this reasonable behaviour (but maybe
> not?).
>
> If that's how it works then it seems reasonable to me.
>
>
Unfortunately this morning's investigation shows this is not the case. When
a recipe is deployed to the target, the on-target script looks for evidence
of this recipe having already been deployed. If that is the case, it
undeploys the previous deploy before deploying this deploy. That is very
good behaviour since it catches the cases where the file list might change
between deployments. But with my tweak as-is, if the user deploys two
packages from the same recipe, the first one will be undeployed and only
the second one will remain deployed; this is not reasonable.

Therefore a v2 will be necessary. The question is, which route do I take?

[-- Attachment #2: Type: text/html, Size: 4018 bytes --]

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Andre McCurdy]

On Fri, Jun 1, 2018 at 7:58 AM, Trevor Woerner <twoerner@gmail.com> wrote:
> On Thu, May 31, 2018 at 11:07 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
>> On Thu, May 31, 2018 at 7:23 PM, Trevor Woerner <twoerner@gmail.com>
>> wrote:
>>
>> I wonder whether the default behaviour should just be to not deploy
>> files from the -dev package?
>>
>
> That would certainly be an improvement. But it still wouldn't make sense to
> copy over the -staticdev or -doc packages either. Maybe a better default
> behaviour would be to not copy over the -dev, -staticdev, -doc, -dbg, and
> -locale packages?

Right. The list of packages to blacklist by default would need some
thought and it's probably more than just the -dev package.

> Or better yet: just copy over the core packages, not any
> of the core-<something> packages? I'm guessing bitbake has a list somewhere
> of these -somethings?

The core packages can be named almost anything, so blacklisting the
packages which shouldn't be deployed is going to be easier than
whitelisting core packages which should be deployed.

> I wouldn't be surprised if the current behaviour exists simply because ${D}
> is already there and ready to go. My tweak extends the behaviour to send
> another already-there directory: ${PKGDEST} + <package>.
>
> If we were to go this route (i.e. being more selective about which packages
> are sent) do you think this should be the new default behaviour, or should
> the existing default behaviour be preserved and a flag used for this tweak?

I was thinking maybe to exclude -dev packages etc by default and
provide a new command line option to over-ride that and deploy
everything (ie a new command line option gets you back to the current
behaviour).

I don't know whether there's a common use case in the devtool workflow
which relies on -dev packages etc being deployed and would be broken
by that change in behaviour though.

>> >> Are the going to be problems if users want to deploy more than one
>> >> package (or switch between a full deploy and a package) and don't
>> >> deploy and undeploy in the right order?
>> >
>> >
>> > In my patch I added a --package option to "devtool deploy-target" but
>> > not to
>> > "devtool undeploy-target". devtool keeps track of what files were
>> > uploaded
>> > by creating a per-package list on the target itself (in /.devtool). When
>> > you
>> > undeploy a recipe, it simply looks for this recipe file on the target,
>> > reads
>> > the files listed there, and undeploys each of the files in that list.
>> >
>> > Therefore, mixing and matching deploys and undeploys amongst recipes
>> > don't
>> > interfere with each other.
>> >
>> > I checked at the time, but I'm not 100% sure now, but I believe if one
>> > deploys two packages from one recipe, the on-target file gets appended.
>> > But
>> > now I'm not 100% sure, so I'll verify this tomorrow (thanks for the
>> > reminder). As such, under my scheme, you can deploy recipes and packages
>> > at
>> > will, but then only undeploy the whole kit and caboodle in one go; given
>> > a
>> > recipe name to undeploy, whatever it finds in the on-target record will
>> > get
>> > undeployed.
>> >
>> > I think any developer will find this reasonable behaviour (but maybe
>> > not?).
>>
>> If that's how it works then it seems reasonable to me.
>>
>
> Unfortunately this morning's investigation shows this is not the case. When
> a recipe is deployed to the target, the on-target script looks for evidence
> of this recipe having already been deployed. If that is the case, it
> undeploys the previous deploy before deploying this deploy. That is very
> good behaviour since it catches the cases where the file list might change
> between deployments. But with my tweak as-is, if the user deploys two
> packages from the same recipe, the first one will be undeployed and only the
> second one will remain deployed; this is not reasonable.
>
> Therefore a v2 will be necessary. The question is, which route do I take?

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Paul Eggleton]

Hi Trevor,

On Friday, 1 June 2018 2:23:59 PM NZST Trevor Woerner wrote:
> Earlier today I was working on recipes for some software which recently
> added the feature to dlopen() one of a couple shared objects based on
> various conditions. But instead of dlopen()'ing the SONAME, they were
> dlopen()'ing the raw .so file. OE's default packaging, of course, added the
> *.so.0 and *.so.0.0.0 files to the ${PN} package, and the raw *.so files to
> the ${PN}-dev package. So when the app tried to run, it failed.
> 
> Investigating, I realised there were two things I could do: 1) add the raw
> *.so files to the ${PN} package (with INSANE_SKIP += "dev-so") or 2) fix
> the app. I decided to try #2, with the help of devtool.
> 
> I made one small change to the app, "devtool deploy-target"'ed it, and,
> AMAZINGLY, everything worked!! Wow, what luck! ... or not.
> 
> Investigating, I discovered that "devtool deploy-target"'s only mode of
> operation is to upload the entire recipe's ${D} directory (i.e.
> ${WORKDIR}/image) to the target! I hope you'll be as surprised by that as
> am I. Considering ${D} includes *.a files, man pages, raw *.so files,
> pkgconfigs, (... thankfully it doesn't also include the -dbg packages) that
> is a lot of stuff to be uploading to the target, a lot of wrong stuff, and
> in my case that hindered my work. Most target filesystems are only 10%
> larger than required to hold their contents, so uploading static libraries,
> man pages, pkgconfigs, etc is not great.

This is intentional. The reason we do it this way is it frees the user from 
having to resolve any packaging issues that might otherwise prevent them from 
actually getting early development code onto the device to test - package QA, 
but also what if important files simply aren't in the right package? The user 
will then have to go and figure out how to fix that before they can get any 
further. devtool is supposed to shorten people's development cycles, 
especially if it's early in development and they tend to be a bit less patient 
to see something working on target.

Now, I appreciate that the flipside of the same coin is that there are issues 
we might pick up as part of package QA that would be helpful to highlight to 
the user (e.g. binary architecture was wrong), so it may be that we should 
revisit this, but I wouldn't want to wholesale switch over to packaging output 
by default - we'd need to be a bit more careful about it.

I will note that a small issue with your patch is that if you do use --package 
you can no longer just use devtool build to build what you need to deploy, 
since that only builds up to do_populate_sysroot / do_packagedata. This in 
turn means that you can't use this option within the eSDK as you can't 
conveniently run bitbake there (again, intentional).

I'm not violently opposed to this patch, but everyone does need to understand 
what the caveats are so we may need some extra notes in the help text at a 
minimum.

> I checked at the time, but I'm not 100% sure now, but I believe if one
> deploys two packages from one recipe, the on-target file gets appended. But
> now I'm not 100% sure, so I'll verify this tomorrow (thanks for the
> reminder). 

The way I recall writing it, it will remove whatever it deployed last time, so 
I would be surprised if it were incremental.

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Trevor Woerner]

Hi Paul,

I was hoping you'd notice and provide feedback :-D

On Tue 2018-06-05 @ 10:59:07 AM, Paul Eggleton wrote:
> > I checked at the time, but I'm not 100% sure now, but I believe if one
> > deploys two packages from one recipe, the on-target file gets appended. But
> > now I'm not 100% sure, so I'll verify this tomorrow (thanks for the
> > reminder). 
> 
> The way I recall writing it, it will remove whatever it deployed last time, so 
> I would be surprised if it were incremental.

Yes, further investigation demonstrated this to be the case. This is a good
feature, since it handles the case where files or filenames might change
between deployments.

I've since updated the patch (locally) so that the on-target filenames of the
deployments, e.g. <recipe>.list and <recipe>.preserve, now include the
optional packagename (if this option is used): e.g. <recipe>.<package>.list
and <recipe>.<package>.preserve. In this way a user can deploy and undeploy
multiple packages from the same recipe independently of each other without any
of them clobbering the other.

The only caveat, now, would be if the user then deployed the entire recipe. In
this case the on-target files would still be the things the user was working
on; the "problem" would be an inconsistency in the on-target devtool
accounting files. In other words, it would be inadvisable to mix and match
between using and not using the --package option within the same recipe.

Best regards,
	Trevor

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Paul Eggleton]

On Tuesday, 5 June 2018 10:59:07 AM NZST Paul Eggleton wrote:
> I will note that a small issue with your patch is that if you do use --
> package 
> you can no longer just use devtool build to build what you need to deploy, 
> since that only builds up to do_populate_sysroot / do_packagedata. This in 
> turn means that you can't use this option within the eSDK as you can't 
> conveniently run bitbake there (again, intentional).

Clearly I'm trying to do too many things this morning - this is completely 
wrong, do_packagedata depends on do_package so there shouldn't be a problem 
here. The rest of my reply stands though.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 5659 bytes --]

On 5/31/18 1:06 PM, Trevor Woerner wrote:
> Instead of installing an entire recipe's build output (i.e. ${D}), allow the
> user to optionally specify a package from said recipe to be installed
> exclusively (i.e. ${PKGDEST}/<package>).
> 

Thinking of developer's workflow, it might be needed to be able to
install debug/dev packages to debug the package on device.

although I think we should be able to segment it and may be have an
option to install the dev part separately

> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---
>  scripts/lib/devtool/deploy.py | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
> index 52e261d560..c022757519 100644
> --- a/scripts/lib/devtool/deploy.py
> +++ b/scripts/lib/devtool/deploy.py
> @@ -169,11 +169,20 @@ def deploy(args, config, basepath, workspace):
>          except Exception as e:
>              raise DevtoolError('Exception parsing recipe %s: %s' %
>                              (args.recipename, e))
> -        recipe_outdir = rd.getVar('D')
> +        if args.package:
> +            recipe_outdir = os.path.join(rd.getVar('PKGDEST'), args.package)
> +        else:
> +            recipe_outdir = rd.getVar('D')
>          if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir):
> -            raise DevtoolError('No files to deploy - have you built the %s '
> -                            'recipe? If so, the install step has not installed '
> -                            'any files.' % args.recipename)
> +            if args.package:
> +                raise DevtoolError('No files to deploy - have you built the %s '
> +                                'package of the %s recipe? If so, the install '
> +                                'step has not installed any files.'
> +                                % (args.package, args.recipename))
> +            else:
> +                raise DevtoolError('No files to deploy - have you built the %s '
> +                                'recipe? If so, the install step has not installed '
> +                                'any files.' % args.recipename)
>  
>          if args.strip and not args.dry_run:
>              # Fakeroot copy to new destination
> @@ -314,8 +323,8 @@ def register_commands(subparsers, context):
>      """Register devtool subcommands from the deploy plugin"""
>  
>      parser_deploy = subparsers.add_parser('deploy-target',
> -                                          help='Deploy recipe output files to live target machine',
> -                                          description='Deploys a recipe\'s build output (i.e. the output of the do_install task) to a live target machine over ssh. By default, any existing files will be preserved instead of being overwritten and will be restored if you run devtool undeploy-target. Note: this only deploys the recipe itself and not any runtime dependencies, so it is assumed that those have been installed on the target beforehand.',
> +                                          help='Deploy build output to a live target machine',
> +                                          description='Deploys either the full recipe\'s build output (i.e. the output of the do_install task) or a package of a recipe to a live target machine over ssh. By default, any existing files will be preserved instead of being overwritten and will be restored if you run devtool undeploy-target. Note: this only deploys the specified item itself and not any runtime dependencies, so it is assumed that those have been installed on the target beforehand.',
>                                            group='testbuild')
>      parser_deploy.add_argument('recipename', help='Recipe to deploy')
>      parser_deploy.add_argument('target', help='Live target machine running an ssh server: user@hostname[:destdir]')
> @@ -325,6 +334,7 @@ def register_commands(subparsers, context):
>      parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
>      parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true')
>      parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
> +    parser_deploy.add_argument('--package', help='Specify a recipe\'s package to deploy', dest='package')
>  
>      strip_opts = parser_deploy.add_mutually_exclusive_group(required=False)
>      strip_opts.add_argument('-S', '--strip',
> @@ -337,8 +347,8 @@ def register_commands(subparsers, context):
>      parser_deploy.set_defaults(func=deploy)
>  
>      parser_undeploy = subparsers.add_parser('undeploy-target',
> -                                            help='Undeploy recipe output files in live target machine',
> -                                            description='Un-deploys recipe output files previously deployed to a live target machine by devtool deploy-target.',
> +                                            help='Undeploy output files from a live target machine',
> +                                            description='Un-deploys output files previously deployed to a live target machine by devtool deploy-target.',
>                                              group='testbuild')
>      parser_undeploy.add_argument('recipename', help='Recipe to undeploy (if not using -a/--all)', nargs='?')
>      parser_undeploy.add_argument('target', help='Live target machine running an ssh server: user@hostname')
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [devtool][PATCH] devtool deploy-target: optionally specify package

[Trevor Woerner]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

Hi Khem,

On Fri, Jun 1, 2018 at 1:11 PM, Khem Raj <raj.khem@gmail.com> wrote:

> On 5/31/18 1:06 PM, Trevor Woerner wrote:
> > Instead of installing an entire recipe's build output (i.e. ${D}), allow
> the
> > user to optionally specify a package from said recipe to be installed
> > exclusively (i.e. ${PKGDEST}/<package>).
> >
>
> Thinking of developer's workflow, it might be needed to be able to
> install debug/dev packages to debug the package on device.
>
> although I think we should be able to segment it and may be have an
> option to install the dev part separately
>

Currently -dev packages will be installed, but -dbg packages will not
(since the components of the -dbg package aren't placed in ${D}). So if
anything this patch (or better yet, the v2 I'll be sending shortly) will
improve the situation.

Although I have noticed that there are differences between the -dbg package
that is created with the normal bitbake workflow and the -dbg package that
is created as part of the "devtool modify ..." workflow.

[-- Attachment #2: Type: text/html, Size: 1531 bytes --]