Re: [OE-core] [PATCH 2/2] oeqa/utils/postactions: transfer whole archive over ssh instead of doing individual copies

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2024-07-05 at 16:46 +0200, Alexis Lothoré via lists.openembedded.org wrote:
> From: Alexis Lothoré <alexis.lothore@bootlin.com>
> 
> Fixes [YOCTO 15536]
> 
> The postactions retrieval actions currently rely on scp executed
> individually on any file or directory expanded from
> TESTIMAGE_FAILED_QA_ARTIFACTS. Unfortunately, symlinks are not preserved
> with this mechanism, which lead to big storage space consumption. Things
> may go even worse if those symlinks create some circular chains. This
> mechanism then needs to be updated to preserve symlinks instead of
> following them during copy. There are multiple ways to do it:
> - create a local archive on the target and execute scp on this file
> - use rsync instead of scp for all files
> - create an archive and pipe it to ssh instead of storing it onto the
>   target
> 
> The first solution may create pressure on targets storage space, while the
> second assumes that rsync is installed on the target, which may not be
> true. So the third one is a compromise: tar is very likely present, at
> least through busybox, and no disk space is used on the target.
> 
> Replace the current per-file scp call by a single call to tar run on the
> target. Retrieve the generated compressed archive directly from SSH output,
> and feed it to another tar process but on host, to uncompress and extract
> it at the same place as before.
> 
> Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> ---
>  meta/lib/oeqa/utils/postactions.py | 19 +++++++++----------
>  1 file changed, 9 insertions(+), 10 deletions(-)
> 
> diff --git a/meta/lib/oeqa/utils/postactions.py b/meta/lib/oeqa/utils/postactions.py
> index ecdddd2d40e3..f7ce25f2af8e 100644
> --- a/meta/lib/oeqa/utils/postactions.py
> +++ b/meta/lib/oeqa/utils/postactions.py
> @@ -62,17 +62,16 @@ def get_artifacts_list(target, raw_list):
>      return result
>  
>  def retrieve_test_artifacts(target, artifacts_list, target_dir):
> +    import io, subprocess
>      local_artifacts_dir = os.path.join(target_dir, "artifacts")
> -    for artifact_path in artifacts_list:
> -        if not os.path.isabs(artifact_path):
> -            bb.warn(f"{artifact_path} is not an absolute path")
> -            continue
> -        try:
> -            dest_dir = os.path.join(local_artifacts_dir, os.path.dirname(artifact_path[1:]))
> -            os.makedirs(dest_dir, exist_ok=True)
> -            target.copyFrom(artifact_path, dest_dir)
> -        except Exception as e:
> -            bb.warn(f"Can not retrieve {artifact_path} from test target: {e}")
> +    try:
> +        cmd = f"tar zcf - {" ".join(artifacts_list)}"

Thanks for the patch. The syntax above causes failures:

https://valkyrie.yoctoproject.org/#/builders/95/builds/59/steps/14/logs/stdio

I've put a fix on master-next which I can squash in but it does make me
wonder how it was tested.

Cheers,

Richard


> +        (status, output) = target.run(cmd, raw = True)
> +        if status != 0 or not output:
> +            raise Exception("Error while fetching compressed artifacts")
> +        p = subprocess.run(["tar", "zxf", "-", "-C", local_artifacts_dir], input=output)
> +    except Exception as e:
> +        bb.warn(f"Can not retrieve {artifact_path} from test target: {e}")
>  
>  def list_and_fetch_failed_tests_artifacts(d, tc):
>      artifacts_list = get_artifacts_list(tc.target, d.getVar("TESTIMAGE_FAILED_QA_ARTIFACTS"))
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201610): https://lists.openembedded.org/g/openembedded-core/message/201610
> Mute This Topic: https://lists.openembedded.org/mt/107054834/1686473
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [richard.purdie@linuxfoundation.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>