Re: [kernelci-members] Working with the KernelCI project

Dmitry Vyukov
2020-09-28T16:24:26Z

On Mon, Sep 28, 2020 at 6:09 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Mon, Sep 28, 2020 at 5:23 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
> > <Nikolai.Kondrashov@redhat.com> wrote:
> > >
> > > Hi Dmitry,
> > >
> > > On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
> > > > The examples are helpful. I think this should be doable. The ability
> > > > to send everything in a single request is handy, and since we don't
> > > > send bulky data (only links), should not cause any issues.
> > >
> > > Awesome!
> > >
> > > > I need an API key, right? How are they distributed? Could you send me
> > > > one to my email privately?
> > >
> > > Yep, just sent it.
> > >
> > > > However, we don't use any Python and use of kcidb-submit will be
> > > > problematic as the app is a Go App Engine app.
> > >
> > > Do you mean you can't run executables, or that having the Python stack in
> > > there as well is not possible/burdensome? Just curious.
> >
> > Appengine is what's called "serverless" today. There are no servers,
> > filesystem, custom binaries. I just upload a set of source Go files
> > and Appengine takes care of the rest. So there is no way to deploy a
> > custom binary written in a different language.
> >
> >
> > > > I looked what's the actual protocol and got to here:
> > > > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
> > > > and this "client" seems to come from google.cloud.pubsub.
> > > >
> > > > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
> > > > it would be much more practical for syzbot to use that directly. But
> > > > what exactly is the protocol? I assume it's that JSON, but is it
> > > > wrapped into some other layers? What's the topic name to publish? Is
> > > > there description/example on this level?
> > >
> > > There's no documentation available for this level, yet, just the code.
> > >
> > > However, yes, it's just JSON submitted to a PubSub topic, currently, and the
> > > schema is exactly as output by "kcidb-schema".
> > >
> > > Please validate JSON fully before submitting to the PubSub topic
> > > ("kcidb-validate" could help you start). We don't have error reporting to
> > > submitters from the server side yet, we'll have to read the logs to find out
> > > what happened, if it breaks :)
> > >
> > > I sent you the project/topic names in the message with the credentials.
> > >
> > > Oh, and we would definitely appreciate a contribution of the Go submission
> > > module, if you make one :)
> > >
> > > > Also, how can I test things? Is there a test instance? Or if I push
> > > > something, can I observe it on the KernelCI dashboard or something?
> > >
> > > Yes, there's a "playground" instance, details are in the message with the
> > > credentials. You can see the results on the "playground" dashboard, and query
> > > them from the "playground" database with "kcidb-query".
> > >
> > > > What would be the procedure for deploying it on the syzbot side?
> > >
> > > Do you mean "deploying the dashboard"?
> >
> > I meant when/how do we deploy this uploading to production. I mean if
> > I do a prototype and it works with a playground, what's next?
> > ... answering my own question, I guess we can make production syzbot
> > upload to the playground kcidb first and then see how it works. If it
> > works well, then we can switch to production kcidb.
> >
> >
> >
> > I have a question re schema. You mentioned "incidents" above. But I
> > don't see them in the kcidb-schema output.
>
> There are no "issues" either...
> Here is what I've got from kcidb-schema (on latest checkout):
> https://gist.githubusercontent.com/dvyukov/a067b63e74399201df921d3846241c82/raw/3f3aa708bbb24f530d49f767a8006f7a51826a80/gistfile1.txt

I've submitted the following json to the pubsub endpoint, and it seems
to succeed as I got server message ID back: 1592574115213643
But I can't find this entry on the dashboard, among other places I've looked at:
https://staging.kernelci.org:3000/d/home/home?orgId=1&from=now-30m&to=now&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&refresh=30m
Wasn't it submitted? Or am I looking in a wrong place? Or is there some log?

{
"builds": [
{
"architecture": "x86_64",
"command": "",
"compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
"config_name": "",
"config_url":
"https://syzkaller.appspot.com/text?tag=KernelConfig\u0026x=d195fe572fb15312",
"description": "",
"duration": 0,
"id": "syzkaller:d195fe572fb15355",
"input_files": null,
"log_url": "",
"misc": null,
"origin": "syzbot",
"output_files": null,
"revision_id": "7ae77150d94d3b535c7b85e6b3647113095e7900",
"start_time": "2020-06-25T03:21:48+00:00",
"valid": true
}
],
"revisions": [
{
"contacts": null,
"description": "",
"discovery_time": "2020-06-25T03:21:48+00:00",
"git_commit_hash": "7ae77150d94d3b535c7b85e6b3647113095e7900",
"git_commit_name": "",
"git_repository_branch": "master",
"git_repository_url":
"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
"id": "7ae77150d94d3b535c7b85e6b3647113095e7900",
"log_url": "",
"message_id": "",
"misc": null,
"origin": "syzbot",
"patch_mboxes": null,
"publishing_time": "",
"tree_name": "",
"valid": true
}
],
"tests": [
{
"build_id": "syzkaller:d195fe572fb15355",
"description": "",
"duration": 0,
"environment": null,
"id": "syzkaller:d195fe572fb15366",
"misc": null,
"origin": "syzbot",
"output_files": [
{
"name": "log.txt",
"url":
"https://syzkaller.appspot.com/text?tag=CrashLog\u0026x=137c5419100000"
},
{
"name": "report.txt",
"url":
"https://syzkaller.appspot.com/text?tag=CrashReport\u0026x=1543f44b100000"
},
{
"name": "repro.syz.txt",
"url":
"https://syzkaller.appspot.com/text?tag=ReproSyz\u0026x=12941c03100000"
},
{
"name": "repro.c",
"url":
"https://syzkaller.appspot.com/text?tag=ReproC\u0026x=12002cf9100000"
}
],
"path": "syzkaller",
"start_time": "2020-06-25T03:21:48+00:00",
"status": "FAIL",
"waived": false
}
],
"version": {
"major": 3,
"minor": 0
}
}