Re: [Buildroot] Flutter-engine package help wanted.

["Yann E. MORIN" <yann.morin.1998@free.fr>]

Adam, All,

On 2023-07-14 20:02 -0700, Adam Duskett spake thusly:
> Hello! I have a task to port Flutter to Buildroot and to make the packages
> suitable for the master branch.
> 
> I have talked with Yann in IRC about the problems facing the first package,
> flutter-engine [ [1]https://github.com/flutter/engine]. However, after another full
> day of hacking away at the package, it's time to talk about a significant
> problem I must overcome.
> 
> Here is a summary of the issue I am facing and the main reason for the email:
> 
> Flutter is developed and maintained by Google.
> Of course, it stands to reason that Flutter does not use a conventional
> configure/build system. Oh no, it couldn't be that easy. Instead, Flutter uses
> gclient and gn from the chromium depot_tools repository, which contains all
> sorts of exciting utilities. One of which is gclient.
> 
> What is gclient? From: [2]https://www.chromium.org/developers/how-tos/depottools
> 
> gclient is a Python script to manage a workspace of modular dependencies that
> are each checked out independently from different subversion or git
> repositories. Features include:
> 
>   - Dependencies can be specified on a per-OS basis.
>   - Dependencies can be specified relative to their parent dependency.
>   - Variables can be used to abstract concepts.
>   - Hooks can be specified to be run after a checkout.
>   - .gclient and DEPS are Python scripts. You can hack in easily or add
>      additional configuration data.
> 
> .gclient file: It's the primary file. It is, in fact, a Python script. It
> specifies the following variables:
> 
>   - solutions: an array of dictionaries specifying the projects that will be
>                fetched.
>   - hooks: additional hooks to be run when this meta checkout is synced.
>   - target_os: an optional array of (target) operating systems to fetch
>                OS-specific dependencies for.
>   - cache_dir: Primarily for bots, multiple working sets use a single git
>                cache.
> 
> Why is this a significant issue?
> 
> The release tarballs from [3]https://github.com/flutter/engine are in no state to
> compile! No, they are only for the use of gclient to download and build a
> source directory structure suitable to build the Flutter engine! If you
> download, extract and attempt to run
> ./tools/gn --no-goma --no-prebuilt-dart-sdk,
> you receive the error message:
> No such file or directory: 'flutter/flutter/third_party/gn/gn.'
> 
> But wait! Wasn't the gn binary just called? No, that's a wrapper in the Flutter
> source tree that formats arguments to call the real gn binary. The real gn is
> not provided in the tarball but is downloaded via gclient
> (among many other repositories.)
> 
> The above leads to the question: What do we do about this? We can't use the
> existing download infrastructure to download the tarball from Git Hub, as that
> isn't meant for us. The gclient download works if called out directly via a
> pre_download hook; however, this leads to other issues in recreating a
> consistent tarball because gclient writes the date to various files all around
> the directory, thus changing the shasum every time I attempt to generate a
> tarball. I can sed the dates in a flutter-engine-gen-tarball script I've made,
> which solves the problem; however, there is one last hurdle to overcome: When I
> compress the directory created by gclient, any .git folder in a subdirectory
> also changes when running mk_tar_gz from the script. The .git directories, of
> course, can be removed, and that DOES create a reproducible tarball, but the
> build system relies on the .git directories to fill in a bunch of information
> before configuring, causing failures.
> 
> I won't ask the best course of action because anything we decide will be pretty
> ugly, thanks to Google and its decisions. Instead, I will ask, what is the
> least-bad solution that can be upstreamed to the master branch?

So, my overall idea was:

    FLUTTER_VERSION = some-hash-or-tag
    FLUTTER_SITE = https://github.com/flutter/engine
    FLUTTER_SITE_METHOD = git   # or download a tarball

    define FLUTTER_POPULATE_WITH_GCLIENT
        rm -rf $(@D)/foo-temp-trash  # in case previous make failed somehow and we needed to restart
        mkdir $(@D)/foo-temp-trash
        tar xf $(FLUTTER_DL_DIR)/$(FLUTTER_SOURCE) -C $(@D)/foo-temp-trash
        cd $(@D)/foo-temp-trash && gclient --some-freaking-options
        . ./support/download/helpers && mk_tar_gz \
            $(@D)/foo-temp-trash \
            flutter-$(FLUTTER_VERSION) \
            $(some-date-which-one-pray-tell) \
            $(FLUTTER_DL_DIR)/flutter-$(FLUTTER_VERSION).br.tar.gz
        rm -rf $(@D)/foo-temp-trash
    endef
    FLUTTER_POST_DOWNLOAD_HOOKS += FLUTTER_POPULATE_WITH_GCLIENT

    FLUTTER_EXTRACT_GCLIENT_STUFF
        # Better use the existing macros that those proper decompressor
        # and tar macro that already strips component; read the code, Luke
        tar xf $(FLUTTER_DL_DIR)/flutter-$(FLUTTER_VERSION).br.tar.gz --strip-component=1 -C $(@D)
    endef
    FLUTTER_POST_EXTRACT_HOOKS += FLUTTER_EXTRACT_GCLIENT_STUFF

And that's basically it. No .hash file because gclient does soe
git-clone, those full repositories are needed at build time, and thus
there is no telling whatstate those local clones will be in (besides a
lot of dates fluctuating...)

Regards,
Yann E. MORIN.

> If you want to play around with any of this, you can clone
> git@gitlab.com:aduskett/buildroot-flutter.git and build the flutter_defconfig.
> 
> Thanks!
> Adam
> 
> Links:
> 1. https://github.com/flutter/engine
> 2. https://www.chromium.org/developers/how-tos/depottools
> 3. https://github.com/flutter/engine

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot