Contribute to KernelCI in Kubernetes or Cloud

Contribute to KernelCI in Kubernetes or Cloud

[Aditya Srivastava]

[-- Attachment #1: Type: text/plain, Size: 188 bytes --]

Hi,
Heard the talk on OSS by Kevin Hilman and khouloud touil , I would like to
help to get KernelCI on K8S or any other help if needed for the same.

Thanks and Regards,
Aditya Srivastava

[-- Attachment #2: Type: text/html, Size: 758 bytes --]

Re: Contribute to KernelCI in Kubernetes or Cloud

[Kevin Hilman]

Hello,

Aditya Srivastava <adityasrivastava301199@gmail.com> writes:

> Heard the talk on OSS by Kevin Hilman and khouloud touil , I would like to
> help to get KernelCI on K8S or any other help if needed for the same.

Thanks for contacting us.  What's your experience with k8s? and how do
you want to help?

As you may have learned in my talk, we do *lots* of kernel builds.

We have Docker images for each arch/toolchain combination[1], and these
are the containers we use in the k8s jobs.

Our pipeline is managed by Jenkins, and you can read a bit about the
Jenkins jobs in the README of the main kernelci-core repo[2].

Where k8s comes in is at the `jenkins/build.jpl` step.  When we get to
this phase, we're ready to do a build for a specific combination of:

- git tree
- git branch
- kernel defconfig
- arch
- compiler

The build step then generates a k8s job (from a jinja2 template) based
on these parameters, and sends it off to a pre-configure k8s cluster
using kubectl.

Here is the github PR[3] that has the jinja2 template and a python
script (gen.py) to generate the k8s job yaml.  There's also another
script (wait.py) which uses the k8s python API to wait for the job to
complete and then fetch the build logs.

I've been the one working on the k8s tooling, but I'm pretty new to it,
so I'm assuming that I made some newbie mistakes and that there are
probably better ways of doing things.

For now, this is already working pretty well and has allowed us to scale
up our build capacity (thanks to compute donations from gcloud and
azure), but any help in this area by experienced engineers would be very
helpful.

I'm particularily interested in your thoughts on improvements on how to
submit/wait for k8s jobs and then fetch the logs.  It's working OK now,
but occaisionaly get problems wher kubectl (or wait.py) gets random
failures, or "connection refused" errors from the k8s cluster.  I cannot
reliably reproduce these reliably but they happen every few thousand
builds.

Anyways, that's a bit of background.  If you have some time to help out,
please have a look at the current expirments and feel free to ask
questions.

It also might be worth your time to join our weekly call.  We meet at
16h UTC every Tuesday via a jitsi video call[4].

Thanks again for you interest and offer to help,

Kevin

[1] https://github.com/kernelci/kernelci-core/tree/master/jenkins/dockerfiles
[2] https://github.com/kernelci/kernelci-core
[3] https://github.com/kernelci/kernelci-core/pull/366/commits/b63da26abc216ad8bc8123d63010a485db2ae3a8
[4] https://meet.kernel.social/kernelci-dev

Re: Contribute to KernelCI in Kubernetes or Cloud

[adityasrivastava301199]

On Mon, Jul 6, 2020 at 11:40 PM Kevin Hilman <khilman@baylibre.com> wrote:

> Hello,
>
> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>
> > Heard the talk on OSS by Kevin Hilman and khouloud touil , I would like
> to
> > help to get KernelCI on K8S or any other help if needed for the same.
>
> Thanks for contacting us.  What's your experience with k8s? and how do
> you want to help?
>
>
Hii Kevin,
I would like to start with apologizing for replying so late. I am sorry.
I was down with a cold and slight fever and pandemic sure gets you worried
over little things.
Then I got a little busy with college stuff.

I am not very experienced with k8s either,
I am working and learning on the way by setting up a monitoring cluster for
OPNFV project.


As you may have learned in my talk, we do *lots* of kernel builds.
>
We have Docker images for each arch/toolchain combination[1], and these
> are the containers we use in the k8s jobs.
>
> Our pipeline is managed by Jenkins, and you can read a bit about the
> Jenkins jobs in the README of the main kernelci-core repo[2].
>
> Where k8s comes in is at the `jenkins/build.jpl` step.  When we get to
> this phase, we're ready to do a build for a specific combination of:
>
> - git tree
> - git branch
> - kernel defconfig
> - arch
> - compiler
>
> The build step then generates a k8s job (from a jinja2 template) based
> on these parameters, and sends it off to a pre-configure k8s cluster
> using kubectl.
>
> Here is the github PR[3] that has the jinja2 template and a python
> script (gen.py) to generate the k8s job yaml.  There's also another
> script (wait.py) which uses the k8s python API to wait for the job to
> complete and then fetch the build logs.
>
> I've been the one working on the k8s tooling, but I'm pretty new to it,
> so I'm assuming that I made some newbie mistakes and that there are
> probably better ways of doing things.
>
> For now, this is already working pretty well and has allowed us to scale
> up our build capacity (thanks to compute donations from gcloud and
> azure), but any help in this area by experienced engineers would be very
> helpful.
>
> I'm particularily interested in your thoughts on improvements on how to
> submit/wait for k8s jobs and then fetch the logs.


I would suggest that for fetching logs we can set up
Elasticsearch-Fluentd-Kibana stack (EFK) or
Elasticsearch-Logstash-Kibana stack (ELK) both work in a similar manner...
They can scale and support HA (high availability).

We can use some alerting system there too, like parse the logs and notify
on failures (which can be
set using alerting rules)

(Although I see kernelci build logs in my mail daily, so that can be
optional)


> It's working OK now,
> but occaisionaly get problems wher kubectl (or wait.py) gets random
> failures, or "connection refused" errors from the k8s cluster.  I cannot

reliably reproduce these reliably but they happen every few thousand
> builds.
>

Yes, I understand.
These stacks I am talking about, can themselves be deployed on a VM or as a
k8s cluster itself.
So it'll be like the cluster is sending the logs to us, not we fetching
them.
Here[1] is my one slide presentation for EFK. I know it is not very well
made but I hope it'll be helpful


(Similarly we can set up a monitoring stack using prometheus - alertmanager
- granfana to collect metrics
like CPU usage, disk space left, memory used,  if you see the need. My work
(not yet complete) can
be seen here[2])


> Anyways, that's a bit of background.  If you have some time to help out,
> please have a look at the current expirments and feel free to ask
> questions.
>
> It also might be worth your time to join our weekly call.  We meet at
> 16h UTC every Tuesday via a jitsi video call[4].
>
> I would love to join next week :D
If not possible, will inform you in advance.


> Thanks again for you interest and offer to help,
>
> Kevin
>
Thank you so much Kevin for giving such a detailed reply.
Again, I did go through the files but I would like to visit them again
before I get back to you with my questions.

>

[1]
> https://github.com/kernelci/kernelci-core/tree/master/jenkins/dockerfiles
> [2] https://github.com/kernelci/kernelci-core
> [3]
> https://github.com/kernelci/kernelci-core/pull/366/commits/b63da26abc216ad8bc8123d63010a485db2ae3a8
> [4] https://meet.kernel.social/kernelci-dev



Thanks and Regards,
Aditya Srivastava

[1] EFK Slide
<https://docs.google.com/presentation/d/1yzOEkCblvojPVhR4MIOOG7bg1k8ACXXX9YluVLGHxNI/edit?usp=sharing>
[2] https://github.com/adi10hero/PAG_K8S

Re: Contribute to KernelCI in Kubernetes or Cloud

[Kevin Hilman]

Aditya Srivastava <adityasrivastava301199@gmail.com> writes:

> On Mon, Jul 6, 2020 at 11:40 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
>> Hello,
>>
>> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>>
>> > Heard the talk on OSS by Kevin Hilman and khouloud touil , I would like
>> to
>> > help to get KernelCI on K8S or any other help if needed for the same.
>>
>> Thanks for contacting us.  What's your experience with k8s? and how do
>> you want to help?
>>
>>
> Hii Kevin,
> I would like to start with apologizing for replying so late. I am sorry.
> I was down with a cold and slight fever and pandemic sure gets you worried
> over little things.
> Then I got a little busy with college stuff.
>
> I am not very experienced with k8s either,
> I am working and learning on the way by setting up a monitoring cluster for
> OPNFV project.
>
>
> As you may have learned in my talk, we do *lots* of kernel builds.
>>
> We have Docker images for each arch/toolchain combination[1], and these
>> are the containers we use in the k8s jobs.
>>
>> Our pipeline is managed by Jenkins, and you can read a bit about the
>> Jenkins jobs in the README of the main kernelci-core repo[2].
>>
>> Where k8s comes in is at the `jenkins/build.jpl` step.  When we get to
>> this phase, we're ready to do a build for a specific combination of:
>>
>> - git tree
>> - git branch
>> - kernel defconfig
>> - arch
>> - compiler
>>
>> The build step then generates a k8s job (from a jinja2 template) based
>> on these parameters, and sends it off to a pre-configure k8s cluster
>> using kubectl.
>>
>> Here is the github PR[3] that has the jinja2 template and a python
>> script (gen.py) to generate the k8s job yaml.  There's also another
>> script (wait.py) which uses the k8s python API to wait for the job to
>> complete and then fetch the build logs.
>>
>> I've been the one working on the k8s tooling, but I'm pretty new to it,
>> so I'm assuming that I made some newbie mistakes and that there are
>> probably better ways of doing things.
>>
>> For now, this is already working pretty well and has allowed us to scale
>> up our build capacity (thanks to compute donations from gcloud and
>> azure), but any help in this area by experienced engineers would be very
>> helpful.
>>
>> I'm particularily interested in your thoughts on improvements on how to
>> submit/wait for k8s jobs and then fetch the logs.
>
>
> I would suggest that for fetching logs we can set up
> Elasticsearch-Fluentd-Kibana stack (EFK) or
> Elasticsearch-Logstash-Kibana stack (ELK) both work in a similar manner...
> They can scale and support HA (high availability).

For now, we are trying to de-couple the build steps from the log
collection and reporting, and for now we're just looking at k8s for
scaling the kernel builds.

On log collection/reporting, just for your info, he last step of the
build is to publish logs and related metadata to the
kernelci-backend[2].  From there we have an existing frontend[2]
(production instance at kernelci.org) that allows basic viewing of
data/logs.  We also have a PoC grafana instance (see the KCIDB tab at
kernelci.org) setup to view some of that data.

> We can use some alerting system there too, like parse the logs and
> notify on failures (which can be set using alerting rules)

This will be part or the reporting side as well, and is part of kcidb[3]

> (Although I see kernelci build logs in my mail daily, so that can be
> optional)
>
>
>> It's working OK now,
>> but occaisionaly get problems wher kubectl (or wait.py) gets random
>> failures, or "connection refused" errors from the k8s cluster.  I cannot
>
> reliably reproduce these reliably but they happen every few thousand
>> builds.
>>
>
> Yes, I understand.
> These stacks I am talking about, can themselves be deployed on a VM or as a
> k8s cluster itself.
> So it'll be like the cluster is sending the logs to us, not we fetching
> them.
> Here[1] is my one slide presentation for EFK. I know it is not very well
> made but I hope it'll be helpful

There have been a couple expeirments already with ELK for the
visualization parts, but they stalled out mainly due to lack of time so
didn't get far enough to show how it could be useful, but we're always
open to examples of how we could better use these tools.

Kevin

[1] https://github.com/kernelci/kernelci-backend
[2] https://github.com/kernelci/kernelci-frontend
[3] https://github.com/kernelci/kcidb

Re: Contribute to KernelCI in Kubernetes or Cloud

[Aditya Srivastava]

On Wed, Jul 15, 2020 at 2:09 PM Aditya Srivastava via groups.io
<adityasrivastava301199=gmail.com@groups.io> wrote:

> On Mon, Jul 6, 2020 at 11:40 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> > Hello,
> >
> > Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
> >
> > > Heard the talk on OSS by Kevin Hilman and khouloud touil , I would like
> > to
> > > help to get KernelCI on K8S or any other help if needed for the same.
> >
> > Thanks for contacting us.  What's your experience with k8s? and how do
> > you want to help?
> >
> >
> Hii Kevin,
> I would like to start with apologizing for replying so late. I am sorry.
> I was down with a cold and slight fever and pandemic sure gets you worried
> over little things.
> Then I got a little busy with college stuff.
>
> I am not very experienced with k8s either,
> I am working and learning on the way by setting up a monitoring cluster for
> OPNFV project.
>
>
> As you may have learned in my talk, we do *lots* of kernel builds.
> >
> We have Docker images for each arch/toolchain combination[1], and these
> > are the containers we use in the k8s jobs.
> >
> > Our pipeline is managed by Jenkins, and you can read a bit about the
> > Jenkins jobs in the README of the main kernelci-core repo[2].
> >
> > Where k8s comes in is at the `jenkins/build.jpl` step.  When we get to
> > this phase, we're ready to do a build for a specific combination of:
> >
> > - git tree
> > - git branch
> > - kernel defconfig
> > - arch
> > - compiler
> >
> > The build step then generates a k8s job (from a jinja2 template) based
> > on these parameters, and sends it off to a pre-configure k8s cluster
> > using kubectl.
> >
> > Here is the github PR[3] that has the jinja2 template and a python
> > script (gen.py) to generate the k8s job yaml.  There's also another
> > script (wait.py) which uses the k8s python API to wait for the job to
> > complete and then fetch the build logs.
> >
> > I've been the one working on the k8s tooling, but I'm pretty new to it,
> > so I'm assuming that I made some newbie mistakes and that there are
> > probably better ways of doing things.
> >
> > For now, this is already working pretty well and has allowed us to scale
> > up our build capacity (thanks to compute donations from gcloud and
> > azure), but any help in this area by experienced engineers would be very
> > helpful.
> >
> > I'm particularily interested in your thoughts on improvements on how to
> > submit/wait for k8s jobs and then fetch the logs.
>
>
> I would suggest that for fetching logs we can set up
> Elasticsearch-Fluentd-Kibana stack (EFK) or
> Elasticsearch-Logstash-Kibana stack (ELK) both work in a similar manner...
> They can scale and support HA (high availability).
>
> We can use some alerting system there too, like parse the logs and notify
> on failures (which can be
> set using alerting rules)
>
> (Although I see kernelci build logs in my mail daily, so that can be
> optional)
>
>
> > It's working OK now,
> > but occaisionaly get problems wher kubectl (or wait.py) gets random
> > failures, or "connection refused" errors from the k8s cluster.  I cannot
>
> reliably reproduce these reliably but they happen every few thousand
> > builds.
> >
>
> Yes, I understand.
> These stacks I am talking about, can themselves be deployed on a VM or as a
> k8s cluster itself.
> So it'll be like the cluster is sending the logs to us, not we fetching
> them.
> Here[1] is my one slide presentation for EFK. I know it is not very well
> made but I hope it'll be helpful
>
>
> (Similarly we can set up a monitoring stack using prometheus - alertmanager
> - granfana to collect metrics
> like CPU usage, disk space left, memory used,  if you see the need. My work
> (not yet complete) can
> be seen here[2])
>
>
> > Anyways, that's a bit of background.  If you have some time to help out,
> > please have a look at the current expirments and feel free to ask
> > questions.
> >
> > It also might be worth your time to join our weekly call.  We meet at
> > 16h UTC every Tuesday via a jitsi video call[4].
> >
> > I would love to join next week :D
> If not possible, will inform you in advance.
>
>
> > Thanks again for you interest and offer to help,
> >
> > Kevin
> >
> Thank you so much Kevin for giving such a detailed reply.
> Again, I did go through the files but I would like to visit them again
> before I get back to you with my questions.
>
> >
>
> [1]
> >
> https://github.com/kernelci/kernelci-core/tree/master/jenkins/dockerfiles
> > [2] https://github.com/kernelci/kernelci-core
> > [3]
> >
> https://github.com/kernelci/kernelci-core/pull/366/commits/b63da26abc216ad8bc8123d63010a485db2ae3a8
> > [4] https://meet.kernel.social/kernelci-dev
>
>
>
> Thanks and Regards,
> Aditya Srivastava
>
> [1] EFK Slide
> <
> https://docs.google.com/presentation/d/1yzOEkCblvojPVhR4MIOOG7bg1k8ACXXX9YluVLGHxNI/edit?usp=sharing
> >
> [2] https://github.com/adi10hero/PAG_K8S
>
> 
>
>


I went through the PRs sent by Kevin, I liked the way gen and wait.py were
implemented.

I had one concern though, in the jinja2 file[1] the way args are inputted,
Can't there be a way to create a config file or something to use instead of
writing that huge args part (which can be error prone)

Thanks and regards,
        Aditya

[1]
https://github.com/kernelci/kernelci-core/pull/366/files#diff-80182b682042b9b2a1a71ac7233563e1R54


---
PS: I wanted to bring it to notice that I am also learning the cloud
orchestration (k8s) part,
      Monitoring and logging on K8S is what I have some experience in,
      My expertise lies in the ML/ DL/ analytics domain.

Re: Contribute to KernelCI in Kubernetes or Cloud

[Guillaume Tucker]

Hi Aditya,

On 17/07/2020 10:40, Aditya Srivastava wrote:
> On Wed, Jul 15, 2020 at 2:09 PM Aditya Srivastava via groups.io
> <adityasrivastava301199=gmail.com@groups.io> wrote:
> 
>> On Mon, Jul 6, 2020 at 11:40 PM Kevin Hilman <khilman@baylibre.com> wrote:
>>
>>> Hello,
>>>
>>> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>>>
>>>> Heard the talk on OSS by Kevin Hilman and khouloud touil , I would like
>>> to
>>>> help to get KernelCI on K8S or any other help if needed for the same.
>>>
>>> Thanks for contacting us.  What's your experience with k8s? and how do
>>> you want to help?
>>>
>>>
>> Hii Kevin,
>> I would like to start with apologizing for replying so late. I am sorry.
>> I was down with a cold and slight fever and pandemic sure gets you worried
>> over little things.
>> Then I got a little busy with college stuff.
>>
>> I am not very experienced with k8s either,
>> I am working and learning on the way by setting up a monitoring cluster for
>> OPNFV project.
>>
>>
>> As you may have learned in my talk, we do *lots* of kernel builds.
>>>
>> We have Docker images for each arch/toolchain combination[1], and these
>>> are the containers we use in the k8s jobs.
>>>
>>> Our pipeline is managed by Jenkins, and you can read a bit about the
>>> Jenkins jobs in the README of the main kernelci-core repo[2].
>>>
>>> Where k8s comes in is at the `jenkins/build.jpl` step.  When we get to
>>> this phase, we're ready to do a build for a specific combination of:
>>>
>>> - git tree
>>> - git branch
>>> - kernel defconfig
>>> - arch
>>> - compiler
>>>
>>> The build step then generates a k8s job (from a jinja2 template) based
>>> on these parameters, and sends it off to a pre-configure k8s cluster
>>> using kubectl.
>>>
>>> Here is the github PR[3] that has the jinja2 template and a python
>>> script (gen.py) to generate the k8s job yaml.  There's also another
>>> script (wait.py) which uses the k8s python API to wait for the job to
>>> complete and then fetch the build logs.
>>>
>>> I've been the one working on the k8s tooling, but I'm pretty new to it,
>>> so I'm assuming that I made some newbie mistakes and that there are
>>> probably better ways of doing things.
>>>
>>> For now, this is already working pretty well and has allowed us to scale
>>> up our build capacity (thanks to compute donations from gcloud and
>>> azure), but any help in this area by experienced engineers would be very
>>> helpful.
>>>
>>> I'm particularily interested in your thoughts on improvements on how to
>>> submit/wait for k8s jobs and then fetch the logs.
>>
>>
>> I would suggest that for fetching logs we can set up
>> Elasticsearch-Fluentd-Kibana stack (EFK) or
>> Elasticsearch-Logstash-Kibana stack (ELK) both work in a similar manner...
>> They can scale and support HA (high availability).
>>
>> We can use some alerting system there too, like parse the logs and notify
>> on failures (which can be
>> set using alerting rules)
>>
>> (Although I see kernelci build logs in my mail daily, so that can be
>> optional)
>>
>>
>>> It's working OK now,
>>> but occaisionaly get problems wher kubectl (or wait.py) gets random
>>> failures, or "connection refused" errors from the k8s cluster.  I cannot
>>
>> reliably reproduce these reliably but they happen every few thousand
>>> builds.
>>>
>>
>> Yes, I understand.
>> These stacks I am talking about, can themselves be deployed on a VM or as a
>> k8s cluster itself.
>> So it'll be like the cluster is sending the logs to us, not we fetching
>> them.
>> Here[1] is my one slide presentation for EFK. I know it is not very well
>> made but I hope it'll be helpful
>>
>>
>> (Similarly we can set up a monitoring stack using prometheus - alertmanager
>> - granfana to collect metrics
>> like CPU usage, disk space left, memory used,  if you see the need. My work
>> (not yet complete) can
>> be seen here[2])
>>
>>
>>> Anyways, that's a bit of background.  If you have some time to help out,
>>> please have a look at the current expirments and feel free to ask
>>> questions.
>>>
>>> It also might be worth your time to join our weekly call.  We meet at
>>> 16h UTC every Tuesday via a jitsi video call[4].
>>>
>>> I would love to join next week :D
>> If not possible, will inform you in advance.
>>
>>
>>> Thanks again for you interest and offer to help,
>>>
>>> Kevin
>>>
>> Thank you so much Kevin for giving such a detailed reply.
>> Again, I did go through the files but I would like to visit them again
>> before I get back to you with my questions.
>>
>>>
>>
>> [1]
>>>
>> https://github.com/kernelci/kernelci-core/tree/master/jenkins/dockerfiles
>>> [2] https://github.com/kernelci/kernelci-core
>>> [3]
>>>
>> https://github.com/kernelci/kernelci-core/pull/366/commits/b63da26abc216ad8bc8123d63010a485db2ae3a8
>>> [4] https://meet.kernel.social/kernelci-dev
>>
>>
>>
>> Thanks and Regards,
>> Aditya Srivastava
>>
>> [1] EFK Slide
>> <
>> https://docs.google.com/presentation/d/1yzOEkCblvojPVhR4MIOOG7bg1k8ACXXX9YluVLGHxNI/edit?usp=sharing
>>>
>> [2] https://github.com/adi10hero/PAG_K8S
>>
>>
>>
>>
> 
> 
> I went through the PRs sent by Kevin, I liked the way gen and wait.py were
> implemented.
> 
> I had one concern though, in the jinja2 file[1] the way args are inputted,
> Can't there be a way to create a config file or something to use instead of
> writing that huge args part (which can be error prone)

Yes, I actually started working on that last week.  It has always
been something to be done, and wasn't a blocker for automated
jobs so it was kept at a lower priority.  But now that logs from
Kubernetes don't filter out tokens on the command line like
Jenkins does, we really need config files for all use-cases.

I"m planning to do it in a similar way as what is in
kernelci-deploy, using configparser in Python.  I can send a link
to the PR when it's ready if you want to review the changes, or
we can discuss that on IRC.

Thanks,
Guillaume


> Thanks and regards,
>         Aditya
> 
> [1]
> https://github.com/kernelci/kernelci-core/pull/366/files#diff-80182b682042b9b2a1a71ac7233563e1R54
> 
> 
> ---
> PS: I wanted to bring it to notice that I am also learning the cloud
> orchestration (k8s) part,
>       Monitoring and logging on K8S is what I have some experience in,
>       My expertise lies in the ML/ DL/ analytics domain.
> 
> 
> 

Re: Contribute to KernelCI in Kubernetes or Cloud

[Kevin Hilman]

Aditya Srivastava <adityasrivastava301199@gmail.com> writes:

[...]

> I had one concern though, in the jinja2 file[1] the way args are inputted,
> Can't there be a way to create a config file or something to use instead of
> writing that huge args part (which can be error prone)

Yes, as Guillaume said, we need a way to use config files for the tools.

However, if we go this route, how do you pass the config file to the k8s
job?  So far, I only know how to start containers with
arguments/parameters passed in env variables.

Kevin

Re: Contribute to KernelCI in Kubernetes or Cloud

[Aditya Srivastava]

On Thu, Jul 23, 2020 at 2:25 AM Kevin Hilman <khilman@baylibre.com> wrote:

> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>
> [...]
>
> > I had one concern though, in the jinja2 file[1] the way args are
> inputted,
> > Can't there be a way to create a config file or something to use instead
> of
> > writing that huge args part (which can be error prone)
>
> Yes, as Guillaume said, we need a way to use config files for the tools.
>
> However, if we go this route, how do you pass the config file to the k8s
> job?  So far, I only know how to start containers with
> arguments/parameters passed in env variables.
>
> Kevin
>

Hi kevin,
You can create a persistent volume and mount the file in the pod.
I can show you an example of how it's done...

If that's not a very high priority task, I can contribute to that part too

But right now my college placements are going on. Final round of placements
is
a hackathon which is this weekend. So I will be able to start with this on
monday.

If its needed ASAP,

Here's the example:
https://github.com/ADI10HERO/PAG_K8S/blob/master/yamls/prometheus-deployment.yaml#L48

And that directory that is  mentioned there is here:
https://github.com/ADI10HERO/PAG_K8S/tree/master/prometheus

So the setup was like:
On a centos VM, repository was cloned on in the /root/

PS: You can use relative paths too....

If an even detailed explanation was expected, please let me know,
I'd be more than happy to write :D

Thanks and regards,
Aditya Srivastava

Re: Contribute to KernelCI in Kubernetes or Cloud

[Kevin Hilman]

Aditya Srivastava <adityasrivastava301199@gmail.com> writes:

> On Thu, Jul 23, 2020 at 2:25 AM Kevin Hilman <khilman@baylibre.com> wrote:
>
>> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>>
>> [...]
>>
>> > I had one concern though, in the jinja2 file[1] the way args are
>> inputted,
>> > Can't there be a way to create a config file or something to use instead
>> of
>> > writing that huge args part (which can be error prone)
>>
>> Yes, as Guillaume said, we need a way to use config files for the tools.
>>
>> However, if we go this route, how do you pass the config file to the k8s
>> job?  So far, I only know how to start containers with
>> arguments/parameters passed in env variables.
>>
>> Kevin
>>
>
> Hi kevin,
> You can create a persistent volume and mount the file in the pod.
> I can show you an example of how it's done...

I know about persistent volumes, but how do you push files to the k8s
cluster so they show up in your volume?

In your example, how does /root/PAG_K8S/prometheus get populated?

Kevin

Re: Contribute to KernelCI in Kubernetes or Cloud

[Aditya Srivastava]

On Thu, Jul 30, 2020 at 10:19:23AM -0700, Kevin Hilman wrote:
> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
> 
> > On Wed, Jul 29, 2020 at 4:32 AM Kevin Hilman <khilman@baylibre.com> wrote:
> >
> >> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
> >>
> >> > On Sat, Jul 25, 2020 at 12:18 AM Kevin Hilman <khilman@baylibre.com>
> >> wrote:
> >> >
> >> >> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
> >> >>
> >> >> > On Thu, Jul 23, 2020 at 2:25 AM Kevin Hilman <khilman@baylibre.com>
> >> >> wrote:
> >> >> >
> >> >> >> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
> >> >> >>
> >> >> >> [...]
> >> >> >>
> >> >> >> > I had one concern though, in the jinja2 file[1] the way args are
> >> >> >> inputted,
> >> >> >> > Can't there be a way to create a config file or something to use
> >> >> instead
> >> >> >> of
> >> >> >> > writing that huge args part (which can be error prone)
> >> >> >>
> >> >> >> Yes, as Guillaume said, we need a way to use config files for the
> >> tools.
> >> >> >>
> >> >> >> However, if we go this route, how do you pass the config file to the
> >> k8s
> >> >> >> job?  So far, I only know how to start containers with
> >> >> >> arguments/parameters passed in env variables.
> >> >> >>
> >> >> >> Kevin
> >> >> >>
> >> >> >
> >> >> > Hi kevin,
> >> >> > You can create a persistent volume and mount the file in the pod.
> >> >> > I can show you an example of how it's done...
> >> >>
> >> >> I know about persistent volumes, but how do you push files to the k8s
> >> >> cluster so they show up in your volume?
> >> >>
> >> >> In your example, how does /root/PAG_K8S/prometheus get populated?
> >> >>
> >> >>
> >> >
> >> > Lets talk with reference to this[1] file
> >> >
> >> > So the volume inside the container at line 41 (A) is the same volume
> >> > (because they have name)
> >> > written under the volume section on line 49. (B)
> >> >
> >> > A : Volume inside the pod / container
> >> > B:  Volume on the host path, i,e a directory ( /root/PAG_K8S/prometheus )
> >> >
> >> > Now root/PAG_K8S/prometheus, is something I need to have already on my
> >> host
> >> > before running the kubectl command
> >> > on the  prometheus-deployment.yaml file.
> >>
> >> So this is the important part.  You already have to have the hostPath on
> >> host running k8s, correct?
> >>
> >>
> > Yes that is correct
> >
> >
> >
> >> Which means if you wanted to use a custom config file for a given job,
> >> you have to push/publish that to the k8s host before running kubectl.
> >>
> >>
> > Yes, in my case I have the files already on the hostPath
> >
> >
> >
> >> > So when I run that file, the volume name which is written under the
> >> > containers section (lines 39 to 43)
> >> > It matches the name of the volume mounts from the ones under the Volumes
> >> > section (lines 46 to 52)
> >> >
> >> >
> >> > And the files /directory(ies) in the volumes section's (volume) paths are
> >> > mounted inside the container
> >> > on the paths mentioned under the volumeMounts section's mountPaths.
> >> >
> >> > For example,
> >> > /root/PAG_K8S/prometheus  (hostpath0) is mounted at  /etc/prometheus
> >> > (hostpath0's path inside the container)
> >> >
> >> > Now /root/PAG_K8S/prometheus in my case doesn't get populated in runtime,
> >> > it has static data....
> >>
> >> Right, so my main question is how would you do this same type of thing
> >> if you have data that needs to be populated at runtime?
> >>
> >> Kevin
> >>
> >
> >
> > 1.  A possibility can be to use config maps, if possible the config map's
> > yaml
> >      can be generated with the needed data at runtime and then it is
> > applied to the pods.
> 
> Ah, I think this is what we need.  I wasn't aware of ConfigMap.
> 
> > 2. We can try looking at helm charts for this purpose, but I am not very
> > familiar with them.
> 
> I'm not familiar at all with helm.
> 
> > I will search more on this, can you give me an example of what is expected
> > ....
> > Or maybe go over the actual  problem or goal if possible.
> >
> > Knowing exactly what is needed will help me refine my search so....
> 
> The overall goal is to get a dynamically generated config file (in
> python configparser, ini-style format) which can be used by some python
> tools in the container.
>

I seee... 

1. So if you have a host Vm on which  there is a k8s pod running
you need to generate the config file on the host VM right?

2. And are these VMs ephemeral?

3. If yes, how do will you generate the config file, I mean even the
_config-map.yaml_ needs to be generated somewhow, right?


> I did a little reading about ConfigMap and it looks like this will do
> the trick.
> 
> Kevin
> 
> 

I hope it works and I will be glad if I helped even a litle.


Is there anything else I could do to contribute? Something very basic
maybe, it can help me get familiar with the project and also help you guys a
little...


Thanks and regards, 
Aditya Srivastava

Re: Contribute to KernelCI in Kubernetes or Cloud

[Kevin Hilman]

Aditya Srivastava <adityasrivastava301199@gmail.com> writes:

> On Thu, Jul 30, 2020 at 10:19:23AM -0700, Kevin Hilman wrote:
>> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>> 
>> > On Wed, Jul 29, 2020 at 4:32 AM Kevin Hilman <khilman@baylibre.com> wrote:
>> >
>> >> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>> >>
>> >> > On Sat, Jul 25, 2020 at 12:18 AM Kevin Hilman <khilman@baylibre.com>
>> >> wrote:
>> >> >
>> >> >> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>> >> >>
>> >> >> > On Thu, Jul 23, 2020 at 2:25 AM Kevin Hilman <khilman@baylibre.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Aditya Srivastava <adityasrivastava301199@gmail.com> writes:
>> >> >> >>
>> >> >> >> [...]
>> >> >> >>
>> >> >> >> > I had one concern though, in the jinja2 file[1] the way args are
>> >> >> >> inputted,
>> >> >> >> > Can't there be a way to create a config file or something to use
>> >> >> instead
>> >> >> >> of
>> >> >> >> > writing that huge args part (which can be error prone)
>> >> >> >>
>> >> >> >> Yes, as Guillaume said, we need a way to use config files for the
>> >> tools.
>> >> >> >>
>> >> >> >> However, if we go this route, how do you pass the config file to the
>> >> k8s
>> >> >> >> job?  So far, I only know how to start containers with
>> >> >> >> arguments/parameters passed in env variables.
>> >> >> >>
>> >> >> >> Kevin
>> >> >> >>
>> >> >> >
>> >> >> > Hi kevin,
>> >> >> > You can create a persistent volume and mount the file in the pod.
>> >> >> > I can show you an example of how it's done...
>> >> >>
>> >> >> I know about persistent volumes, but how do you push files to the k8s
>> >> >> cluster so they show up in your volume?
>> >> >>
>> >> >> In your example, how does /root/PAG_K8S/prometheus get populated?
>> >> >>
>> >> >>
>> >> >
>> >> > Lets talk with reference to this[1] file
>> >> >
>> >> > So the volume inside the container at line 41 (A) is the same volume
>> >> > (because they have name)
>> >> > written under the volume section on line 49. (B)
>> >> >
>> >> > A : Volume inside the pod / container
>> >> > B:  Volume on the host path, i,e a directory ( /root/PAG_K8S/prometheus )
>> >> >
>> >> > Now root/PAG_K8S/prometheus, is something I need to have already on my
>> >> host
>> >> > before running the kubectl command
>> >> > on the  prometheus-deployment.yaml file.
>> >>
>> >> So this is the important part.  You already have to have the hostPath on
>> >> host running k8s, correct?
>> >>
>> >>
>> > Yes that is correct
>> >
>> >
>> >
>> >> Which means if you wanted to use a custom config file for a given job,
>> >> you have to push/publish that to the k8s host before running kubectl.
>> >>
>> >>
>> > Yes, in my case I have the files already on the hostPath
>> >
>> >
>> >
>> >> > So when I run that file, the volume name which is written under the
>> >> > containers section (lines 39 to 43)
>> >> > It matches the name of the volume mounts from the ones under the Volumes
>> >> > section (lines 46 to 52)
>> >> >
>> >> >
>> >> > And the files /directory(ies) in the volumes section's (volume) paths are
>> >> > mounted inside the container
>> >> > on the paths mentioned under the volumeMounts section's mountPaths.
>> >> >
>> >> > For example,
>> >> > /root/PAG_K8S/prometheus  (hostpath0) is mounted at  /etc/prometheus
>> >> > (hostpath0's path inside the container)
>> >> >
>> >> > Now /root/PAG_K8S/prometheus in my case doesn't get populated in runtime,
>> >> > it has static data....
>> >>
>> >> Right, so my main question is how would you do this same type of thing
>> >> if you have data that needs to be populated at runtime?
>> >>
>> >> Kevin
>> >>
>> >
>> >
>> > 1.  A possibility can be to use config maps, if possible the config map's
>> > yaml
>> >      can be generated with the needed data at runtime and then it is
>> > applied to the pods.
>> 
>> Ah, I think this is what we need.  I wasn't aware of ConfigMap.
>> 
>> > 2. We can try looking at helm charts for this purpose, but I am not very
>> > familiar with them.
>> 
>> I'm not familiar at all with helm.
>> 
>> > I will search more on this, can you give me an example of what is expected
>> > ....
>> > Or maybe go over the actual  problem or goal if possible.
>> >
>> > Knowing exactly what is needed will help me refine my search so....
>> 
>> The overall goal is to get a dynamically generated config file (in
>> python configparser, ini-style format) which can be used by some python
>> tools in the container.
>>
>
> I seee... 
>
> 1. So if you have a host Vm on which  there is a k8s pod running
> you need to generate the config file on the host VM right?

We don't specifically need a config file on the host, just in the
container.

> 2. And are these VMs ephemeral?

Yes.  We are using dynamically scaling VMs in both GCE and Azure clouds.
Due to cost, we have access to lots of VMs, but they are preemptible,
spot intstances so they can (and will) disappear randomly.

> 3. If yes, how do will you generate the config file, I mean even the
> _config-map.yaml_ needs to be generated somewhow, right?

I'm not sure yet since I haven't looked closely at the ConfigMap, but I
had the impression that this could be generated as parf of the k8s job
yaml.

>> I did a little reading about ConfigMap and it looks like this will do
>> the trick.
>> 
>> Kevin
>> 
>> 
>
> I hope it works and I will be glad if I helped even a litle.
>
>
> Is there anything else I could do to contribute? Something very basic
> maybe, it can help me get familiar with the project and also help you guys a
> little...

Yes.

One area we're starting to look at is how to best use static volumes in
each cluster.

Linux kernel builds can be made more efficient by using ccache to cache
build results and make subsequent builds faster.  But due to our
ephemeral VMs, the best way to do that would be to have a static volume
that is shared across all pods in a cluster.

I know this is possible, but I have yet to look into that.  If you have
some ideas on how to update our k8s job template[1] to use static
volumes, that would be very useful.

just FYI, I'll be OoO all of next week, but will be back on 10 Aug for
any follow-up.

Thanks again for you interest in helping,

Kevin

[1] https://github.com/kernelci/kernelci-core/blob/master/templates/k8s/job-build.jinja2