Online tool for playing with variable expansion - first steps and where to host it

Online tool for playing with variable expansion - first steps and where to host it

[chris.laplante]

[-- Attachment #1: Type: text/plain, Size: 3061 bytes --]

Hi all,

This bug ("A web based playground for experimenting with bitbake variable expansion") has been preying on my mind ever since I read it on Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14805.

I'm happy to say that I have a simple proof-of-concept implementation built using Pyodide. The code needs some cleanup before I feel comfortable sharing it, but here are some details:


  1.  Consumes an unmodified version of BitBake source code (currently 2.0.16).
  2.  Plays some games with the Python import system (https://docs.python.org/3/reference/import.html) to shim out certain modules (currently just fcntl) that BitBake imports but Pyodide doesn't provide. (*1)
  3.  Executes the following test using JavaScript (*2):

        pyodide.runPython(`
            from bb.data_smart import DataSmart
        `)

        let DataSmart = pyodide.globals.get('DataSmart');
        const d = DataSmart();

        d.setVar("A", "B");
        d.setVar("A:test", "C");
        d.setVar("OVERRIDES", "test");

        console.log(d.getVar("A"));

        DataSmart.destroy();


As far as next steps, I was thinking I would build a simple-ish web app using https://github.com/microsoft/monaco-editor (the editor that powers VS Code). This would let us implement a Rust Playground-type of experience.

The two main questions in my mind are where to keep the code and where to host it?

For the code, I lean towards a separate repository. I was just going to put it under my company's GitHub: https://github.com/Agilent. This is what I'll do unless people feel strongly otherwise. (I'd rather not put it in BitBake itself since due to CORS restrictions I'll probably need to keep tar.gz copies of BitBake in the repo so the client can fetch it (*3). I also don't think we want all the webpack and TypeScript clutter in BitBake)

For hosting, assuming it goes on our GitHub, I was just going to use GitHub pages. The first iteration will be a client-side only application (*4)(*5), so no server will be required except a static endpoint for serving BitBake source code as mentioned above.

Does anyone have strong objections to this plan?

Notes:
    (*1) Interestingly Pyodide already provides sqlite3: https://til.simonwillison.net/python/sqlite-in-pyodide
    (*2) Exposing DataSmart to JavaScript is not necessarily relevant for the online tool; I just thought it was cool. The actual tool will probably just call `pyodide.runPython` with the user-supplied Python code.
    (*3) I foresee bundling a couple of copies of BitBake - at the very least, one with the old overrides syntax and a modern copy. That way people can play with both.
    (*4) In the future, I can envision a server-side component for shimming things like AUTOREV so that we can get git revisions; maybe also tie in https://github.com/petersalomonsen/wasm-git.
(*5) If we wanted to get really crazy, could also potentially leverage https://github.com/binji/wasm-clang - 'bitbake -c compile' in browser, anyone?

Thanks,
Chris


[-- Attachment #2: Type: text/html, Size: 14863 bytes --]

Re: [bitbake-devel] Online tool for playing with variable expansion - first steps and where to host it

[Richard Purdie]

Hi,

On Wed, 2024-02-14 at 16:59 +0000, Chris Laplante via
lists.openembedded.org wrote:
> This bug (“A web based playground for experimenting with bitbake variable 
> expansion”) has been preying on my mind ever since I read it on 
> Bugzilla:https://bugzilla.yoctoproject.org/show_bug.cgi?id=14805.


This does sound quite neat and I'm happy to see people experimenting
like this.

[...]

> As far as next steps, I was thinking I would build a simple-ish web app using 
> https://github.com/microsoft/monaco-editor (the editor that powers VS Code). 
> This would let us implement a Rust Playground-type of experience.
>  
> The two main questions in my mind are where to keep the code and where to host it?
>  
> For the code, I lean towards a separate repository. I was just going to put it 
> under my company’s GitHub:https://github.com/Agilent. This is what I’ll do unless
> people feel strongly otherwise. (I’d rather not put it in BitBake itself since due 
> to CORS restrictions I’ll probably need to keep tar.gz copies of BitBake in the 
> repo so the client can fetch it (*3). I also don’t think we want all the webpack 
> and TypeScript clutter in BitBake)

I agree it doesn't belong in bitbake itself. I did wonder about
something like the yoctoproject github if you wanted more exposure
there but I don't have a strong preference either way. 

> For hosting, assuming it goes on our GitHub, I was just going to use GitHub pages.
>  The first iteration will be a client-side only application (*4)(*5), so no server 
> will be required except a static endpoint for serving BitBake source code as mentioned above.
>  
> Does anyone have strong objections to this plan?

No objection, I mainly wanted to reply to show someone read it and
thought it sounds fun :)

Cheers,

Richard

RE: [bitbake-devel] Online tool for playing with variable expansion - first steps and where to host it

[chris.laplante]

> > I'll do unless people feel strongly otherwise. (I'd rather not put it
> > in BitBake itself since due to CORS restrictions I'll probably need to
> > keep tar.gz copies of BitBake in the repo so the client can fetch it
> > (*3). I also don't think we want all the webpack and TypeScript
> > clutter in BitBake)
> 
> I agree it doesn't belong in bitbake itself. I did wonder about something like the
> yoctoproject github if you wanted more exposure there but I don't have a strong
> preference either way.

I also don't have a strong preference. I envision eventually I'll add little "Run" links in the BitBake manual next to code samples, especially in the chapter on variable overrides, which will take the reader directly to the playground. So I think no matter where we host it, people will eventually find it. 

> 
> No objection, I mainly wanted to reply to show someone read it and thought it
> sounds fun :)

Thank you :)! 

One more question - what to name it? I was tentatively calling it bitodide, but I kind of hate the name. I think "BitBake Playground" is the obvious choice. Could also do "BitBake Explorer" (ala Compiler Explorer). But I think I like Playground better...

Chris

Re: [bitbake-devel] Online tool for playing with variable expansion - first steps and where to host it

[Richard Purdie]

On Thu, 2024-02-15 at 15:29 +0000, chris.laplante@agilent.com wrote:
> > > I'll do unless people feel strongly otherwise. (I'd rather not put it
> > > in BitBake itself since due to CORS restrictions I'll probably need to
> > > keep tar.gz copies of BitBake in the repo so the client can fetch it
> > > (*3). I also don't think we want all the webpack and TypeScript
> > > clutter in BitBake)
> > 
> > I agree it doesn't belong in bitbake itself. I did wonder about something like the
> > yoctoproject github if you wanted more exposure there but I don't have a strong
> > preference either way.
> 
> I also don't have a strong preference. I envision eventually I'll add little "Run" 
> links in the BitBake manual next to code samples, especially in the chapter on 
> variable overrides, which will take the reader directly to the playground. So 
> I think no matter where we host it, people will eventually find it. 

That does potentially change things a little as if we want to make
links from the official project docs, putting it somewhere with our
other project repos does start to make sightly more sense.

I say that as people do move on and do other things and we'd have no
way to recover other than changing the links in that case, which we can
only do with the newer versions, not retrospectively.

> > No objection, I mainly wanted to reply to show someone read it and thought it
> > sounds fun :)
> 
> Thank you :)! 
> 
> One more question - what to name it? I was tentatively calling it bitodide, 
> but I kind of hate the name. I think "BitBake Playground" is the obvious 
> choice. Could also do "BitBake Explorer" (ala Compiler Explorer). But I think 
> I like Playground better...

I prefer Playground fwiw.

Cheers,

Richard

RE: [bitbake-devel] Online tool for playing with variable expansion - first steps and where to host it

[chris.laplante]

> > I also don't have a strong preference. I envision eventually I'll add
> > little "Run" links in the BitBake manual next to code samples,
> > especially in the chapter on variable overrides, which will take the
> > reader directly to the playground. So I think no matter where we host it,
> people will eventually find it.
> 
> That does potentially change things a little as if we want to make links from the
> official project docs, putting it somewhere with our other project repos does
> start to make sightly more sense.
> 
> I say that as people do move on and do other things and we'd have no way to
> recover other than changing the links in that case, which we can only do with
> the newer versions, not retrospectively.

Works for me. Who do I talk to about getting a repo set up on Yocto's GitHub (or wherever)?

Thanks,
Chris

Re: [bitbake-devel] Online tool for playing with variable expansion - first steps and where to host it

[Richard Purdie]

On Thu, 2024-02-15 at 17:25 +0000, chris.laplante@agilent.com wrote:
> > > I also don't have a strong preference. I envision eventually I'll add
> > > little "Run" links in the BitBake manual next to code samples,
> > > especially in the chapter on variable overrides, which will take the
> > > reader directly to the playground. So I think no matter where we host it,
> > people will eventually find it.
> > 
> > That does potentially change things a little as if we want to make links from the
> > official project docs, putting it somewhere with our other project repos does
> > start to make sightly more sense.
> > 
> > I say that as people do move on and do other things and we'd have no way to
> > recover other than changing the links in that case, which we can only do with
> > the newer versions, not retrospectively.
> 
> Works for me. Who do I talk to about getting a repo set up on Yocto's GitHub 
> (or wherever)?

We ask Michael Halstead (copied) who will need your github username and
the name of the repo we want to create.

I'm supposed to agree this with the YP TSC too I think so I'll forward
this to them and check they don't object.

Cheers,

Richard

RE: [bitbake-devel] Online tool for playing with variable expansion - first steps and where to host it

[chris.laplante]

> > Works for me. Who do I talk to about getting a repo set up on Yocto's
> > GitHub (or wherever)?
> 
> We ask Michael Halstead (copied) who will need your github username and the
> name of the repo we want to create.
> 
> I'm supposed to agree this with the YP TSC too I think so I'll forward this to
> them and check they don't object.


As for the name of the repo, I think just bitbake_playground or something like that, capitalized and/or punctuated as desired.

Thanks,
Chris

BB Datastore Playground preview is now live

[chris.laplante]

Hi all,
 
> This bug ("A web based playground for experimenting with bitbake variable
> expansion") has been preying on my mind ever since I read it on Bugzilla:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14805.
> 
> I'm happy to say that I have a simple proof-of-concept implementation built
> using Pyodide. The code needs some cleanup before I feel comfortable sharing
> it, but here are some details:

It took all Saturday, but a very perfunctory version of this tool is now live :). Check it out here: https://yoctoproject.github.io/bb-datastore-playground. Many thanks to Michael for setting up the GitHub repo for me!

It is basically a Python REPL with added 'bb' goodness. 

It is very rough - no styling on the page, and it doesn't work on mobile. But it is already useful so please give it a try. It requires a modern-ish browser (though I haven't categorized what this actually means).

Next steps are to add a Godbolt-like editor and 'Run' button.

Thanks,
Chris

Re: BB Datastore Playground preview is now live

[Michael Halstead]

[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]

On Mon, May 13, 2024 at 7:55 AM chris.laplante@agilent.com <
chris.laplante@agilent.com> wrote:

> Hi all,
>
> > This bug ("A web based playground for experimenting with bitbake variable
> > expansion") has been preying on my mind ever since I read it on Bugzilla:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14805.
> >
> > I'm happy to say that I have a simple proof-of-concept implementation
> built
> > using Pyodide. The code needs some cleanup before I feel comfortable
> sharing
> > it, but here are some details:
>
> It took all Saturday, but a very perfunctory version of this tool is now
> live :). Check it out here:
> https://yoctoproject.github.io/bb-datastore-playground. Many thanks to
> Michael for setting up the GitHub repo for me!
>
> It is basically a Python REPL with added 'bb' goodness.
>
> It is very rough - no styling on the page, and it doesn't work on mobile.
> But it is already useful so please give it a try. It requires a modern-ish
> browser (though I haven't categorized what this actually means).
>

The interpreter works for me in Chrome and Firefox on Fedora 40. ;) It
seems to handle weird whitespace fairly well. Both copy-paste and arrow
keys are working as expected too.

> Next steps are to add a Godbolt-like editor and 'Run' button.
>
> Thanks,
> Chris
>


-- 
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engineer

[-- Attachment #2: Type: text/html, Size: 2655 bytes --]

RE: BB Datastore Playground preview is now live

[chris.laplante]

>> It is very rough - no styling on the page, and it doesn't work on mobile. But it is
>> already useful so please give it a try. It requires a modern-ish browser (though I
>> haven't categorized what this actually means).
> 
> The interpreter works for me in Chrome and Firefox on Fedora 40. ;) It seems to
> handle weird whitespace fairly well. Both copy-paste and arrow keys are
> working as expected too.

Thanks for testing :). I cannot claim credit for that robustness though. All praise there should go to jQuery Terminal which I am using underneath the covers ;). https://terminal.jcubic.pl/

Chris