KCIDB: Support more checkout data

KCIDB: Support more checkout data

[Nikolai Kondrashov]

Hello again, everyone (potentially) involved with sending data to KCIDB,

The previous schema update (v4.4, test data enhancements) is already being
deployed, but I have prepared more backwards-compatible changes to go out with
v4.5.

This time I rolled everything into a single commit, but I'll split it up, if
we have disagreements, or need to work more on the changes.

The PR is at https://github.com/kernelci/kcidb-io/pull/87

But I'll repeat the commit message here:

Add support for three more checkout fields: `git_commit_tags`,
`git_commit_message`, and `git_repository_branch_tip`.

The `git_commit_tags` is an array of strings representing annotated tags
pointing directly at the commit being checked out, as seen in the source
repository. I.e. the output of `git tag --points-at <commit>`. Set to an empty
array, if the commit has no tags.

The `git_commit_message` is intended to hold the complete message of the
commit being checked out, *both* subject and body. I.e. the output of `git
show -s --format=%B`. We're putting the subject and the body together, as it's
quite easy to extract the subject in SQL, while full-text search is easier and
more efficient to do over a single column.

Finally, the `git_repository_branch_tip` is a boolean flag, which should be
set to `true`, when the commit being checked out is at the tip of the branch
at the moment of the checkout (as specified in `start_time`). Essentially, if
you're always testing only the tip of the branch, you can set this to `true`
unconditionally. This flag would let us extract the checkouts which
represented the branch state over time, and produce a rough history of branch
changes, which we can then use for (regression) analysis and graphs, in lieu
of actual commit graph walking.

The (abbreviated) schema for all three fields is below and is very simple:

    "git_commit_tags": {
        "type": "array",
        "items": {"type": "string"},
    },
    "git_commit_message": {"type": "string"},
    "git_repository_branch_tip": {"type": "boolean"}

The PR has the complete schema, with inline docs.

Tell me what you think!
Nick

Re: KCIDB: Support more checkout data

[Nikolai Kondrashov]

On 8/21/24 8:02 PM, Nikolai Kondrashov wrote:
> Hello again, everyone (potentially) involved with sending data to KCIDB,
> 
> The previous schema update (v4.4, test data enhancements) is already being
> deployed, but I have prepared more backwards-compatible changes to go out with
> v4.5.
> 
> This time I rolled everything into a single commit, but I'll split it up, if
> we have disagreements, or need to work more on the changes.
> 
> The PR is at https://github.com/kernelci/kcidb-io/pull/87
> 
> But I'll repeat the commit message here:
> 
> Add support for three more checkout fields: `git_commit_tags`,
> `git_commit_message`, and `git_repository_branch_tip`.
> 
> The `git_commit_tags` is an array of strings representing annotated tags
> pointing directly at the commit being checked out, as seen in the source
> repository. I.e. the output of `git tag --points-at <commit>`. Set to an empty
> array, if the commit has no tags.
> 
> The `git_commit_message` is intended to hold the complete message of the
> commit being checked out, *both* subject and body. I.e. the output of `git
> show -s --format=%B`. We're putting the subject and the body together, as it's
> quite easy to extract the subject in SQL, while full-text search is easier and
> more efficient to do over a single column.
> 
> Finally, the `git_repository_branch_tip` is a boolean flag, which should be
> set to `true`, when the commit being checked out is at the tip of the branch
> at the moment of the checkout (as specified in `start_time`). Essentially, if
> you're always testing only the tip of the branch, you can set this to `true`
> unconditionally. This flag would let us extract the checkouts which
> represented the branch state over time, and produce a rough history of branch
> changes, which we can then use for (regression) analysis and graphs, in lieu
> of actual commit graph walking.
> 
> The (abbreviated) schema for all three fields is below and is very simple:
> 
>     "git_commit_tags": {
>         "type": "array",
>         "items": {"type": "string"},
>     },
>     "git_commit_message": {"type": "string"},
>     "git_repository_branch_tip": {"type": "boolean"}
> 
> The PR has the complete schema, with inline docs.

I'm going to merge this and start work on the support on Wednesday, Aug 28, if
there are no objections by that time.

Nick

Re: KCIDB: Support more checkout data

[Nikolai Kondrashov]

On 8/21/24 8:02 PM, Nikolai Kondrashov wrote:
> Hello again, everyone (potentially) involved with sending data to KCIDB,
> 
> The previous schema update (v4.4, test data enhancements) is already being
> deployed, but I have prepared more backwards-compatible changes to go out with
> v4.5.

Support for this has been deployed last week.

Please consider sending the `git_repository_branch_tip` boolean field, if you
can, as it will help us move forward on regression detection.

The commit messages would help us make dashboards more useful, and tags would
help us track release history.

Thank you!
Nick