Re: Contributing ARM tests results to KCIDB

Nikolai Kondrashov
2020-09-18T16:57:24Z

On 9/18/20 7:42 PM, Cristian Marussi wrote:
> Hi Nick,
>
> On Fri, Sep 18, 2020 at 06:53:28PM +0300, Nikolai Kondrashov wrote:
>> On 9/18/20 6:30 PM, Nikolai Kondrashov wrote:
>>> Yes, I think it's one of the problems you uncovered :)
>>>
>>> The schema allows for fully-compliant RFC3339 timestamps, but the BigQuery
>>> database on the backend doesn't understand some of them. In particular it
>>> doesn't understand the date-only timestamps you send. E.g. "2020-09-13".
>>> That's what I wanted to fix today, but ran out of time.
>>
>> Looking at this more it seems that Python's jsonschema module simply doesn't
>> enforce the requirements we put on those fields 🤦. You can send essentially
>> what you want and then hit BigQuery, which is serious about them.
>
> ...in fact on my side I check too with jsonschema in my script before using kcidb :D

Taking a peek into jsonschema's code it seems it should support verifying
that, but perhaps you need to enable it explicitly?

>>
>> Sorry about that.
>>
>
> No worries.
>
>> I opened an issue for this: https://github.com/kernelci/kcidb/issues/108
>>
>> For now please just make sure your timestamp comply with RFC3339.
>>
>> You can produce such a timestamp e.g. using "date --rfc-3339=s".
>
> I'll anyway fix my data on my side too, to have the real discovery timestamp.

Great! And glad to hear it worked for you :)

Have a nice weekend!
Nick

>>
>> Nick
>>
>
> Thanks
>
> Cristian
>
>> On 9/18/20 6:30 PM, Nikolai Kondrashov wrote:
>>> On 9/18/20 6:21 PM, Cristian Marussi wrote:
>>> > So in order to carry on my experiments, I've just tried to push a new dataset
>>> > with a few changes in my data-layout to mimic what I see other origins do; this
>>> > contained something like 38 builds across 4 different revisions (with brand new
>>> > revisions IDs), but I cannot see anything on the UI: I just keep seeing the old
>>> > push from yesterday.
>>> >
>>> > JSON seems valid and kcidb-submit does not report any error even using -l DEBUG.
>>> > (I pushed >30mins ago)
>>> >
>>> > Any idea ?
>>>
>>> Yes, I think it's one of the problems you uncovered :)
>>>
>>> The schema allows for fully-compliant RFC3339 timestamps, but the BigQuery
>>> database on the backend doesn't understand some of them. In particular it
>>> doesn't understand the date-only timestamps you send. E.g. "2020-09-13".
>>> That's what I wanted to fix today, but ran out of time.
>>>
>>> Additionally, the backend doesn't have a way to report a problem to the
>>> submitter at the moment. We intend to fix that, but for now it's possible only
>>> through us looking at the logs and sending a message to the submitter :)
>>>
>>> To work around this you can pad your timestamps with dummy date and time
>>> data.
>>>
>>> E.g. instead of sending:
>>>
>>> 2020-09-13
>>>
>>> you can send:
>>>
>>> 2020-09-13 00:00:00+00:00
>>>
>>> Hopefully that's the only problem. It could be, since you managed to send data
>>> before :)
>>>
>>> Nick
>>>
>>> On 9/18/20 6:21 PM, Cristian Marussi wrote:
>>> > Hi Nikolai,
>>> >
>>> > On Thu, Sep 17, 2020 at 08:26:15PM +0300, Nikolai Kondrashov wrote:
>>> >> On 9/17/20 7:22 PM, Cristian Marussi wrote:
>>> >>> It works too ... :D
>>> >>>
>>> >>> https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=arm:2020-07-07:d3d7689c2cc9503266cac3bc777bb4ddae2e5f2e
>>> >>
>>> >> Whoa, awesome!
>>> >>
>>> >> And you have already uncovered a few issues we need to fix, too!
>>> >> I will deal with them tomorrow.
>>> >>
>>> >>> ..quick question though....given that now I'll have to play quite a bit
>>> >>> with it and see how's better to present our data, if anythinjg missing etc etc,
>>> >>> is there any chance (or way) that if I submmit the same JSON report multiple
>>> >>> times with slight differences here and there (but with the same IDs clearly)
>>> >>> I'll get my DB updated in the bits I have changed: as an example I've just
>>> >>> resubmitted the same report with added discovery_time and descriptions, and got
>>> >>> NO errors, but I cannot see the changes in the UI (unless they have still to
>>> >>> propagate...)..or maybe I can obtain the same effect by dropping my dataset
>>> >>> before re-submitting ?
>>> >>
>>> >> Right now it's not supported (with various possible quirks if attempted).
>>> >> So, preferably, submit only one, complete and final instance of each object
>>> >> (with unique ID) for now.
>>> >>
>>> >> We have a plan to support merging missing properties across multiple reported
>>> >> objects with the same ID.
>>> >>
>>> >> Object A Object B Dashboard/Notifications
>>> >>
>>> >> FieldX: Foo Foo Foo
>>> >> FieldY: Bar Bar
>>> >> FieldZ: Baz Baz
>>> >> FieldU: Red Blue Red/Blue
>>> >>
>>> >> Since we're using a distributed database we cannot really maintain order
>>> >> (without introducing artificial global lock), so the order of the reports
>>> >> doesn't matter. We can only guarantee that a present value would override
>>> >> missing value. It would be undefined which value would be picked among
>>> >> multiple different values.
>>> >>
>>> >> This would allow gradual reporting of each object, but no editing, sorry.
>>> >>
>>> >> However, once again, this is a plan with some research done, only.
>>> >> I plan to start implementing it within a few weeks.
>>> >>
>>> >
>>> > So in order to carry on my experiments, I've just tried to push a new dataset
>>> > with a few changes in my data-layout to mimic what I see other origins do; this
>>> > contained something like 38 builds across 4 different revisions (with brand new
>>> > revisions IDs), but I cannot see anything on the UI: I just keep seeing the old
>>> > push from yesterday.
>>> >
>>> > JSON seems valid and kcidb-submit does not report any error even using -l DEBUG.
>>> > (I pushed >30mins ago)
>>> >
>>> > Any idea ?
>>> >
>>> > Thanks
>>> >
>>> > Cristian
>>> >
>>> >> Nick
>>> >>
>>> >> On 9/17/20 7:22 PM, Cristian Marussi wrote:
>>> >>> On Thu, Sep 17, 2020 at 04:52:30PM +0300, Nikolai Kondrashov wrote:
>>> >>>> Hi Christian,
>>> >>>>
>>> >>>> On 9/17/20 3:50 PM, Cristian Marussi wrote:
>>> >>>>> Hi Nikolai,
>>> >>>>>
>>> >>>>> I work at ARM in the Kernel team and, in short, we'd like certainly to
>>> >>>>> contribute our internal Kernel test results to KCIDB.
>>> >>>>
>>> >>>> Wonderful!
>>> >>>>
>>> >>>>> After having attended your LPC2020 TestMC and KernelCI/BoF, I've now cooked
>>> >>>>> up some KCIDB JSON test report (seemingly valid against your KCIDB v3 schema)
>>> >>>>> and I'd like to start experimenting with kci-submit (on non-production
>>> >>>>> instances), so as to assess how to fit our results into your schema and maybe
>>> >>>>> contribute with some new KCIDB requirements if strictly needed.
>>> >>>>
>>> >>>> Great, this is exactly what we need, welcome aboard :)
>>> >>>>
>>> >>>> Please don't hesitate to reach out on kernelci@groups.io or on #kernelci on
>>> >>>> freenode.net, if you have any questions, problems, or requirements.
>>> >>>>
>>> >>>>> Is it possible to get some valid credentials and a playground instance to
>>> >>>>> point at ?
>>> >>>>
>>> >>>> Absolutely, I created credentials for you and sent them in a separate message.
>>> >>>>
>>> >>>> You can use origin "arm" for the start, unless you have multiple CI systems
>>> >>>> and want to differentiate them somehow in your reports.
>>> >>>>
>>> >>>> Nick
>>> >>>>
>>> >>> Thanks !
>>> >>>
>>> >>> It works too ... :D
>>> >>>
>>> >>> https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=arm:2020-07-07:d3d7689c2cc9503266cac3bc777bb4ddae2e5f2e
>>> >>>
>>> >>> ..quick question though....given that now I'll have to play quite a bit
>>> >>> with it and see how's better to present our data, if anythinjg missing etc etc,
>>> >>> is there any chance (or way) that if I submmit the same JSON report multiple
>>> >>> times with slight differences here and there (but with the same IDs clearly)
>>> >>> I'll get my DB updated in the bits I have changed: as an example I've just
>>> >>> resubmitted the same report with added discovery_time and descriptions, and got
>>> >>> NO errors, but I cannot see the changes in the UI (unless they have still to
>>> >>> propagate...)..or maybe I can obtain the same effect by dropping my dataset
>>> >>> before re-submitting ?
>>> >>>
>>> >>> Regards
>>> >>>
>>> >>> Thanks
>>> >>>
>>> >>> Cristian
>>> >>>
>>> >>>> On 9/17/20 3:50 PM, Cristian Marussi wrote:
>>> >>>>> Hi Nikolai,
>>> >>>>>
>>> >>>>> I work at ARM in the Kernel team and, in short, we'd like certainly to
>>> >>>>> contribute our internal Kernel test results to KCIDB.
>>> >>>>>
>>> >>>>> After having attended your LPC2020 TestMC and KernelCI/BoF, I've now cooked
>>> >>>>> up some KCIDB JSON test report (seemingly valid against your KCIDB v3 schema)
>>> >>>>> and I'd like to start experimenting with kci-submit (on non-production
>>> >>>>> instances), so as to assess how to fit our results into your schema and maybe
>>> >>>>> contribute with some new KCIDB requirements if strictly needed.
>>> >>>>>
>>> >>>>> Is it possible to get some valid credentials and a playground instance to
>>> >>>>> point at ?
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>>
>>> >>>>> Regards
>>> >>>>>
>>> >>>>> Cristian
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> >
>>> >
>>> > >
>>> >
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>
>
>
>