KCIDB contribuition #KCIDB

KCIDB contribuition #KCIDB

[Santiago.Esteban via info]

Good morning KernelCI, Nikolai,

As I mention here before, I work at Microchip, and we are building a 
test farm based on Labgrid which currently, allow us to test Linux 
kernels in several ARM SoC devices (Cortex-A5, ARM9 variants). We would 
like to contribute to KernelCI effort with our builds/tests but support 
for non Lava external labs is not available yet.

Therefore, It seems that KCIDB is the place for us. We would like to 
start contributing our builds/tests to this project.

As part of our current efforts, we have build a local instance of the 
KernelCI infrastructure. I've already asked Nicolai @DevConf, but here, 
at a wider audience, Are there any tool that allows to submmit  
"standard" KernelCI formatted data into the KCIDB database?


Best regards,

Santiago Esteban

ps. Nikolai, I liked a lot your presentation @DevConf, it helped me to 
understand some aspect of KernelCI  and KCIDB to me.

Re: KCIDB contribuition #KCIDB

[Nikolai Kondrashov]

Hello Santiago,

On 2/22/21 10:02 AM, Santiago.Esteban via info via groups.io wrote:
> Good morning KernelCI, Nikolai,
> 
> As I mention here before, I work at Microchip, and we are building a
> test farm based on Labgrid which currently, allow us to test Linux
> kernels in several ARM SoC devices (Cortex-A5, ARM9 variants). We would
> like to contribute to KernelCI effort with our builds/tests but support
> for non Lava external labs is not available yet.
> 
> Therefore, It seems that KCIDB is the place for us. We would like to
> start contributing our builds/tests to this project.
> 
> As part of our current efforts, we have build a local instance of the
> KernelCI infrastructure. I've already asked Nicolai @DevConf, but here,
> at a wider audience, Are there any tool that allows to submmit
> "standard" KernelCI formatted data into the KCIDB database?

I suppose if you already drive everything with an instance of KernelCI
infrastructure, you can just use the KCIDB interface code that was merged
there. However, I think Guillaume would be able to help you better here.

Guillaume, do you think that would work?

> Best regards,
> 
> Santiago Esteban
> 
> ps. Nikolai, I liked a lot your presentation @DevConf, it helped me to
> understand some aspect of KernelCI  and KCIDB to me.

Ah, glad to hear that, thanks :)

Nick

Re: KCIDB contribuition #KCIDB

[Guillaume Tucker]

On 22/02/2021 09:41, Nikolai Kondrashov wrote:
> Hello Santiago,
> 
> On 2/22/21 10:02 AM, Santiago.Esteban via info via groups.io wrote:
>> Good morning KernelCI, Nikolai,
>>
>> As I mention here before, I work at Microchip, and we are building a
>> test farm based on Labgrid which currently, allow us to test Linux
>> kernels in several ARM SoC devices (Cortex-A5, ARM9 variants). We would
>> like to contribute to KernelCI effort with our builds/tests but support
>> for non Lava external labs is not available yet.
>>
>> Therefore, It seems that KCIDB is the place for us. We would like to
>> start contributing our builds/tests to this project.
>>
>> As part of our current efforts, we have build a local instance of the
>> KernelCI infrastructure. I've already asked Nicolai @DevConf, but here,
>> at a wider audience, Are there any tool that allows to submmit
>> "standard" KernelCI formatted data into the KCIDB database?
> 
> I suppose if you already drive everything with an instance of KernelCI
> infrastructure, you can just use the KCIDB interface code that was merged
> there. However, I think Guillaume would be able to help you better here.
> 
> Guillaume, do you think that would work?

The production instance of kernelci.org is not submitting data to
KCIDB yet because of the overhead related to opening BigQuery
connections.  Until the work to use the streaming interface with
a persistent connection is implemented, and potentially a more
longer-term implementation is in place (i.e. not doing it in
kernelci-backend), I would not recommend doing that.

However, if you are available to help it would be great to speed
up the work with redesigning kernelci-backend.  It should have
happened about a year ago but there have been many distractions
on the way slowing things down.  It's becoming the main sticking
point now, so it will soon be receiving the attention it
deserves.

One thing I don't quite get, is that if you already have an
instance of KernelCI driving your test lab, why not have your
test lab connected to the kernelci.org instance in the same way?
Having Labgrid properly supported by KernelCI is still on the
roadmap, in fact it's related to the redesign of
kernelci-backend.  

Having your own CI pipeline is mostly useful if you want to build
your own kernels or run tests in a private environment.  All the
current KCIDB contributions are coming from preexisting CI
systems which do just that (distro kernels, private test labs).
But if you want to use the builds from kernelci.org and do what
all the other labs on kernelci.org already do, then using KCIDB
directly doesn't seem like the logical choice to me (and results
going to kernelci.org will also end up in KCIDB).

Best wishes,
Guillaume

Re: KCIDB contribuition #KCIDB

[Nikolai Kondrashov]

On 2/22/21 12:25 PM, Guillaume Tucker wrote:
> On 22/02/2021 09:41, Nikolai Kondrashov wrote:
>> I suppose if you already drive everything with an instance of KernelCI
>> infrastructure, you can just use the KCIDB interface code that was merged
>> there. However, I think Guillaume would be able to help you better here.
>>
>> Guillaume, do you think that would work?
> 
> The production instance of kernelci.org is not submitting data to
> KCIDB yet because of the overhead related to opening BigQuery
> connections.  Until the work to use the streaming interface with
> a persistent connection is implemented, and potentially a more
> longer-term implementation is in place (i.e. not doing it in
> kernelci-backend), I would not recommend doing that.

Yeah, the current implementation would be bogged down by the amount of
results KernelCI production is producing. However, Santiago, how many
builds/tests do you see produced per day? Perhaps it's worth a try
to see how that's handled, and maybe it's not too bad? Meanwhile
Michal could perhaps finish the rework needed for faster streaming?

Nick

Re: KCIDB contribuition #KCIDB

[Santiago.Esteban via info]

On 22/2/21 11:31, Nikolai Kondrashov wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know 
> the content is safe
>
> On 2/22/21 12:25 PM, Guillaume Tucker wrote:
>> On 22/02/2021 09:41, Nikolai Kondrashov wrote:
>>> I suppose if you already drive everything with an instance of KernelCI
>>> infrastructure, you can just use the KCIDB interface code that was 
>>> merged
>>> there. However, I think Guillaume would be able to help you better 
>>> here.
>>>
>>> Guillaume, do you think that would work?
>>
>> The production instance of kernelci.org is not submitting data to
>> KCIDB yet because of the overhead related to opening BigQuery
>> connections.  Until the work to use the streaming interface with
>> a persistent connection is implemented, and potentially a more
>> longer-term implementation is in place (i.e. not doing it in
>> kernelci-backend), I would not recommend doing that.
>
> Yeah, the current implementation would be bogged down by the amount of
> results KernelCI production is producing. However, Santiago, how many
> builds/tests do you see produced per day? Perhaps it's worth a try
> to see how that's handled, and maybe it's not too bad? Meanwhile
> Michal could perhaps finish the rework needed for faster streaming?
>
> Nick
>
Our CI right now focus on daily builds of  linux-next and linux-mainline 
repositories with only two targets (sama5_defconf, at91_dt_defconf): 4 
builds tested in 7 boards (and two compilers) = 56 results daily.

I guess, not a lot, considering the amount of data that bigger CI's 
farms produce.

BR,

Santi

Re: KCIDB contribuition #KCIDB

[Nikolai Kondrashov]

On 2/24/21 11:42 AM, Santiago.Esteban via info via groups.io wrote:
> On 22/2/21 11:31, Nikolai Kondrashov wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know
>> the content is safe
>>
>> On 2/22/21 12:25 PM, Guillaume Tucker wrote:
>>> On 22/02/2021 09:41, Nikolai Kondrashov wrote:
>>>> I suppose if you already drive everything with an instance of KernelCI
>>>> infrastructure, you can just use the KCIDB interface code that was
>>>> merged
>>>> there. However, I think Guillaume would be able to help you better
>>>> here.
>>>>
>>>> Guillaume, do you think that would work?
>>>
>>> The production instance of kernelci.org is not submitting data to
>>> KCIDB yet because of the overhead related to opening BigQuery
>>> connections.  Until the work to use the streaming interface with
>>> a persistent connection is implemented, and potentially a more
>>> longer-term implementation is in place (i.e. not doing it in
>>> kernelci-backend), I would not recommend doing that.
>>
>> Yeah, the current implementation would be bogged down by the amount of
>> results KernelCI production is producing. However, Santiago, how many
>> builds/tests do you see produced per day? Perhaps it's worth a try
>> to see how that's handled, and maybe it's not too bad? Meanwhile
>> Michal could perhaps finish the rework needed for faster streaming?
>>
>> Nick
>>
> Our CI right now focus on daily builds of  linux-next and linux-mainline
> repositories with only two targets (sama5_defconf, at91_dt_defconf): 4
> builds tested in 7 boards (and two compilers) = 56 results daily.
> 
> I guess, not a lot, considering the amount of data that bigger CI's
> farms produce.

Yeah, that's not much, and you could probably manage with the current
KCIDB-submission code in kernelci. However, if you're going to have your
lab connected anyway soon, the setup effort might not be worth it.

Nick

Re: KCIDB contribuition #KCIDB

[Santiago.Esteban via info]

On 22/2/21 11:25, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 22/02/2021 09:41, Nikolai Kondrashov wrote:
>> Hello Santiago,
>>
>> On 2/22/21 10:02 AM, Santiago.Esteban via info via groups.io wrote:
>>> Good morning KernelCI, Nikolai,
>>>
>>> As I mention here before, I work at Microchip, and we are building a
>>> test farm based on Labgrid which currently, allow us to test Linux
>>> kernels in several ARM SoC devices (Cortex-A5, ARM9 variants). We would
>>> like to contribute to KernelCI effort with our builds/tests but support
>>> for non Lava external labs is not available yet.
>>>
>>> Therefore, It seems that KCIDB is the place for us. We would like to
>>> start contributing our builds/tests to this project.
>>>
>>> As part of our current efforts, we have build a local instance of the
>>> KernelCI infrastructure. I've already asked Nicolai @DevConf, but here,
>>> at a wider audience, Are there any tool that allows to submmit
>>> "standard" KernelCI formatted data into the KCIDB database?
>> I suppose if you already drive everything with an instance of KernelCI
>> infrastructure, you can just use the KCIDB interface code that was merged
>> there. However, I think Guillaume would be able to help you better here.
>>
>> Guillaume, do you think that would work?
> The production instance of kernelci.org is not submitting data to
> KCIDB yet because of the overhead related to opening BigQuery
> connections.  Until the work to use the streaming interface with
> a persistent connection is implemented, and potentially a more
> longer-term implementation is in place (i.e. not doing it in
> kernelci-backend), I would not recommend doing that.
>
> However, if you are available to help it would be great to speed
> up the work with redesigning kernelci-backend.  It should have
> happened about a year ago but there have been many distractions
> on the way slowing things down.  It's becoming the main sticking
> point now, so it will soon be receiving the attention it
> deserves.
>
> One thing I don't quite get, is that if you already have an
> instance of KernelCI driving your test lab, why not have your
> test lab connected to the kernelci.org instance in the same way?
> Having Labgrid properly supported by KernelCI is still on the
> roadmap, in fact it's related to the redesign of
> kernelci-backend.
> Having your own CI pipeline is mostly useful if you want to build
> your own kernels or run tests in a private environment.  All the
> current KCIDB contributions are coming from preexisting CI
> systems which do just that (distro kernels, private test labs).
> But if you want to use the builds from kernelci.org and do what
> all the other labs on kernelci.org already do, then using KCIDB
> directly doesn't seem like the logical choice to me (and results
> going to kernelci.org will also end up in KCIDB).

Currently, we build linux-mainline and linux-next along with our 
repositories linux4sam (https://github.com/linux4sam). We also use our 
Labgrid farm to tests embedded distributions (Buildroot, Yocto and 
Openwrt). Soon (I hope), we will include more test for other parts of 
our software stack (u-boot, at91bootstrap) and other hardware tests in 
our devices/platforms. Our farm contains 7 different boards today, and 
it will keep growing.

If all results from KernelCI will end up also in KCIDB (this was not 
clear to me), obviously, it does not make sense push my tests in both 
databases.  As you know, we can push test results to staging (I'm not 
allowed to push my builds), but we do not have means to get notified 
when a new builds are available for testing.  That's what is missing 
(for Labgrid or other external lab). While we work in this front, I 
understood, from Nikolai's last week presentation, that it would be nice 
to push also our result to KCIDB too (and it would be fairly easy). 
Anyway, for me it is clear now, that connecting my farm to KernelCI is 
key, and as a bonus, we'll get KCIDB :-).

>
> Best wishes,
> Guillaume

BR,

Santi