Enabling TOML support for user settings

Enabling TOML support for user settings

[Jeny Sheth]

Hello everyone,

We have enabled TOML support for user settings in KernelCI-Core.
Python package `toml` has been integrated to parse the settings file. 
Please see the Github issue[1] for more details on this transition.

The sample config file `kernelci.toml.sample`[2] has been added to the 
repository.
As per the former convention with `.conf`, we will continue having a 
`DEFAULT` section and other sections for different command line tools in 
the `.toml` file.
For example:
```
[DEFAULT]

storage = "http://localhost:5002"
db_config = "localhost"
api_config = "docker-host"

[kci_build]

build_config = "next"
arch="x86"
j = 3
```

With TOML, we can use different data types for values (strings with 
double quotes, integers, float). Unlike `configparser` this removes the 
need for data type conversion for values.

Components such as database, labs, and storage will follow the 
convention of the form ["storage:docker-host"].
For the correct parsing from TOML decoder, we need to use double quotes 
around the name when it includes ":".
Below is the example for more detail:
```
["lab:my-lava-lab"]

user = "user-name"
lab_token = "1234-5678"

["db:staging.kernelci.org"]

db_token = "XXX-XXXX-XXXX-XXXX"
api = "https://api.staging.kernelci.org/"
```

Another suggestion is to use dotted keys instead of ":" for the above 
section names. That way we can remove double quotes around the section 
name. However, that can cause issues when we have dotted values e.g. 
"staging.kernelci.org".
If we go with the dotted name, it will look like 
[db.staging.kernelci.org] and the parser will assume that it is a nested 
dictionary.

For example:
```
[db.staging.kernelci.org]
api = "https://api.staging.kernelci.org/"

```
Above will be parsed as:
```
'db': {'staging': {'kernelci': {'org': {'api': 
'https://api.staging.kernelci.org/'}}}}
```


To overcome that,  we can use double quotes around the value for dotted 
notation.
Something like this:
```
[db."staging.kernelci.org"]
api = "https://api.staging.kernelci.org/"
```

That will be parsed as:
```
'db': {'staging.kernelci.org': {'api': 'https://api.staging.kernelci.org/'}}
```

We would like your comments/suggestions to select the best approach 
possible here for naming conventions. Please check out the discussion[3] 
here and feel free to provide your input.

The `kernelci-pipeline` has already started adapting to this new format. 
This is the work in progress[4] for reference.

Standard locations for the file are the same as `kernelci.conf` file i.e.
- kernelci.toml in the current working directory
- ~/.config/kernelci/kernelci.toml for per-user settings
- /etc/kernelci/kernelci.toml for system-wide settings

Along with this new format, we are still keeping support for the old 
`kernelci.conf` for now. However, It is recommended to switch from the 
`.conf` to `.toml` as the older format will eventually be deprecated.
We are planning to support the older format until we move from the old 
backend to the new API. As per the planning, the transition will take 
place by the end of 2023. Suggestions for deciding the deprecation 
timeline are welcome.


Thanks,
Jeny

[1] https://github.com/kernelci/kernelci-core/issues/1913
[2] https://github.com/kernelci/kernelci-core/blob/main/kernelci.toml.sample
[3] 
https://github.com/kernelci/kernelci-core/issues/1913#issuecomment-1572217965
[4] https://github.com/kernelci/kernelci-pipeline/pull/266/

Re: Enabling TOML support for user settings

[Guillaume Tucker]

Hi Jeny,

On 06/06/2023 09:12, Jeny Sheth wrote:
> Hello everyone,
> 
> We have enabled TOML support for user settings in KernelCI-Core.
> Python package `toml` has been integrated to parse the settings file. Please see the Github issue[1] for more details on this transition.
> 
> The sample config file `kernelci.toml.sample`[2] has been added to the repository.
> As per the former convention with `.conf`, we will continue having a `DEFAULT` section and other sections for different command line tools in the `.toml` file.
> For example:
> ```
> [DEFAULT]
> 
> storage = "http://localhost:5002"
> db_config = "localhost"
> api_config = "docker-host"
> 
> [kci_build]
> 
> build_config = "next"
> arch="x86"
> j = 3
> ```
> 
> With TOML, we can use different data types for values (strings with double quotes, integers, float). Unlike `configparser` this removes the need for data type conversion for values.
> 
> Components such as database, labs, and storage will follow the convention of the form ["storage:docker-host"].
> For the correct parsing from TOML decoder, we need to use double quotes around the name when it includes ":".
> Below is the example for more detail:
> ```
> ["lab:my-lava-lab"]

It's worth noting that "lab" configs have been renamed
to "runtime" now.  But this is not related to the move to TOML,
just to keep examples consistent with the YAML config.

> user = "user-name"
> lab_token = "1234-5678"
> 
> ["db:staging.kernelci.org"]
> 
> db_token = "XXX-XXXX-XXXX-XXXX"
> api = "https://api.staging.kernelci.org/"
> ```
> 
> Another suggestion is to use dotted keys instead of ":" for the above section names. That way we can remove double quotes around the section name. However, that can cause issues when we have dotted values e.g. "staging.kernelci.org".
> If we go with the dotted name, it will look like [db.staging.kernelci.org] and the parser will assume that it is a nested dictionary.
> 
> For example:
> ```
> [db.staging.kernelci.org]
> api = "https://api.staging.kernelci.org/"
> 
> ```
> Above will be parsed as:
> ```
> 'db': {'staging': {'kernelci': {'org': {'api': 'https://api.staging.kernelci.org/'}}}}
> ```
> 
> 
> To overcome that,  we can use double quotes around the value for dotted notation.
> Something like this:
> ```
> [db."staging.kernelci.org"]
> api = "https://api.staging.kernelci.org/"
> ```
> 
> That will be parsed as:
> ```
> 'db': {'staging.kernelci.org': {'api': 'https://api.staging.kernelci.org/'}}
> ```
> 
> We would like your comments/suggestions to select the best approach possible here for naming conventions. Please check out the discussion[3] here and feel free to provide your input.

Thanks for detailing the changes.  I would personally be in
favour of relying on the built-in namespacing feature of TOML
with dotted keys as that would simplify our code base and remove
scope for bugs or corner cases.  The likelihood of users hitting
the issue of creating a key such as api.staging.kernelci.org
rather than api."staging.kernelci.org" can be easily avoided with
a sample settings file and documentation examples.  Also this is
a standard thing in TOML, and it's a widespread config language
standard now so I would expect many users to already be aware of
this.

> The `kernelci-pipeline` has already started adapting to this new format. This is the work in progress[4] for reference.
>
> Standard locations for the file are the same as `kernelci.conf` file i.e.
> - kernelci.toml in the current working directory
> - ~/.config/kernelci/kernelci.toml for per-user settings
> - /etc/kernelci/kernelci.toml for system-wide settings
> 
> Along with this new format, we are still keeping support for the old `kernelci.conf` for now. However, It is recommended to switch from the `.conf` to `.toml` as the older format will eventually be deprecated.

In particular, the new LAVA callback handler service now
exclusively relies on a TOML settings file to be present in order
to find credentials for uploading files to storage (job logs...).

All the regular command line tools and new pipeline services
support both the legacy .conf settings and the new TOML format.

> We are planning to support the older format until we move from the old backend to the new API. As per the planning, the transition will take place by the end of 2023. Suggestions for deciding the deprecation timeline are welcome.

Please note that the official timeline for deprecating the legacy
system hasn't been confirmed yet, but yes in principle it
shouldn't happen earlier than the end of 2023.  So roughly
speaking, users should move to TOML settings by the end of the
year.  A formal timeline with more details will be shared later
when things are more certain.

Thanks,
Guillaume

> [1] https://github.com/kernelci/kernelci-core/issues/1913
> [2] https://github.com/kernelci/kernelci-core/blob/main/kernelci.toml.sample
> [3] https://github.com/kernelci/kernelci-core/issues/1913#issuecomment-1572217965
> [4] https://github.com/kernelci/kernelci-pipeline/pull/266/

Re: Enabling TOML support for user settings

[Jeny Sheth]

Hello,

Thank you Guillaume for your suggestions and comments.


Followed by our discussions with other community members in the KernelCI
community meeting, we decided to use TOML dotted keys as it is a standard way
of defining sub-tables in TOML.

The syntax will look like "section-name.section-config".


For example,
```
[db.docker-host]
     api ="http://localhost:8001"
```


In case the section config name includes "." (dot) in the name,
it must be surrounded by "" (double-quotes) to parse it properly with
TOML decoder.

For example,
```
[db."staging.kernelci.org"]

api ="https://api.staging.kernelci.org/"
```


Please look at these pull requests for further clarification:
1.https://github.com/kernelci/kernelci-core/pull/2033
2.https://github.com/kernelci/kernelci-pipeline/pull/288
3.https://github.com/kernelci/kernelci-pipeline/pull/289

Please note that the old `.conf` setting file will continue to support ":" separator for
section names during the deprecation period.


Thank you,
Jeny

On 16/06/23 5:06 pm, Guillaume Tucker wrote:
> Hi Jeny,
>
> On 06/06/2023 09:12, Jeny Sheth wrote:
>> Hello everyone,
>>
>> We have enabled TOML support for user settings in KernelCI-Core.
>> Python package `toml` has been integrated to parse the settings file. Please see the Github issue[1] for more details on this transition.
>>
>> The sample config file `kernelci.toml.sample`[2] has been added to the repository.
>> As per the former convention with `.conf`, we will continue having a `DEFAULT` section and other sections for different command line tools in the `.toml` file.
>> For example:
>> ```
>> [DEFAULT]
>>
>> storage = "http://localhost:5002"
>> db_config = "localhost"
>> api_config = "docker-host"
>>
>> [kci_build]
>>
>> build_config = "next"
>> arch="x86"
>> j = 3
>> ```
>>
>> With TOML, we can use different data types for values (strings with double quotes, integers, float). Unlike `configparser` this removes the need for data type conversion for values.
>>
>> Components such as database, labs, and storage will follow the convention of the form ["storage:docker-host"].
>> For the correct parsing from TOML decoder, we need to use double quotes around the name when it includes ":".
>> Below is the example for more detail:
>> ```
>> ["lab:my-lava-lab"]
> It's worth noting that "lab" configs have been renamed
> to "runtime" now.  But this is not related to the move to TOML,
> just to keep examples consistent with the YAML config.
>
>> user = "user-name"
>> lab_token = "1234-5678"
>>
>> ["db:staging.kernelci.org"]
>>
>> db_token = "XXX-XXXX-XXXX-XXXX"
>> api = "https://api.staging.kernelci.org/"
>> ```
>>
>> Another suggestion is to use dotted keys instead of ":" for the above section names. That way we can remove double quotes around the section name. However, that can cause issues when we have dotted values e.g. "staging.kernelci.org".
>> If we go with the dotted name, it will look like [db.staging.kernelci.org] and the parser will assume that it is a nested dictionary.
>>
>> For example:
>> ```
>> [db.staging.kernelci.org]
>> api = "https://api.staging.kernelci.org/"
>>
>> ```
>> Above will be parsed as:
>> ```
>> 'db': {'staging': {'kernelci': {'org': {'api': 'https://api.staging.kernelci.org/'}}}}
>> ```
>>
>>
>> To overcome that,  we can use double quotes around the value for dotted notation.
>> Something like this:
>> ```
>> [db."staging.kernelci.org"]
>> api = "https://api.staging.kernelci.org/"
>> ```
>>
>> That will be parsed as:
>> ```
>> 'db': {'staging.kernelci.org': {'api': 'https://api.staging.kernelci.org/'}}
>> ```
>>
>> We would like your comments/suggestions to select the best approach possible here for naming conventions. Please check out the discussion[3] here and feel free to provide your input.
> Thanks for detailing the changes.  I would personally be in
> favour of relying on the built-in namespacing feature of TOML
> with dotted keys as that would simplify our code base and remove
> scope for bugs or corner cases.  The likelihood of users hitting
> the issue of creating a key such as api.staging.kernelci.org
> rather than api."staging.kernelci.org" can be easily avoided with
> a sample settings file and documentation examples.  Also this is
> a standard thing in TOML, and it's a widespread config language
> standard now so I would expect many users to already be aware of
> this.
>
>> The `kernelci-pipeline` has already started adapting to this new format. This is the work in progress[4] for reference.
>>
>> Standard locations for the file are the same as `kernelci.conf` file i.e.
>> - kernelci.toml in the current working directory
>> - ~/.config/kernelci/kernelci.toml for per-user settings
>> - /etc/kernelci/kernelci.toml for system-wide settings
>>
>> Along with this new format, we are still keeping support for the old `kernelci.conf` for now. However, It is recommended to switch from the `.conf` to `.toml` as the older format will eventually be deprecated.
> In particular, the new LAVA callback handler service now
> exclusively relies on a TOML settings file to be present in order
> to find credentials for uploading files to storage (job logs...).
>
> All the regular command line tools and new pipeline services
> support both the legacy .conf settings and the new TOML format.
>
>> We are planning to support the older format until we move from the old backend to the new API. As per the planning, the transition will take place by the end of 2023. Suggestions for deciding the deprecation timeline are welcome.
> Please note that the official timeline for deprecating the legacy
> system hasn't been confirmed yet, but yes in principle it
> shouldn't happen earlier than the end of 2023.  So roughly
> speaking, users should move to TOML settings by the end of the
> year.  A formal timeline with more details will be shared later
> when things are more certain.
>
> Thanks,
> Guillaume
>
>> [1] https://github.com/kernelci/kernelci-core/issues/1913
>> [2] https://github.com/kernelci/kernelci-core/blob/main/kernelci.toml.sample
>> [3] https://github.com/kernelci/kernelci-core/issues/1913#issuecomment-1572217965
>> [4] https://github.com/kernelci/kernelci-pipeline/pull/266/