RFC: CCC Linux Kernel SIG

RFC: CCC Linux Kernel SIG

[Dan Middleton]

Hi from the CCC!

Many of our companies also collaborate in the Confidential Computing 
Consortium.
There we are proposing a SIG and invite your feedback and involvement:

https://github.com/confidential-computing/governance/pull/196

The entirety of the proposal is also copied below.

Thanks,

Dan Middleton
CCC TAC Chair


# Proposal: CCC Linux Kernel SIG

This proposal seeks to accelerate Confidential Computing feature 
velocity in the Linux kernel.
By creating a Linux Kernel SIG in the CCC we can foster community dialog 
to develop vendor neutral
architectural approaches to new features and thereby reduce complexity 
in associated kernel patches.


# Background:

One of the technical limitations to the adoption rate for Confidential 
Computing is the speed at
which we as a community can add features to the Linux kernel. Each 
vendor has similar needs of the
kernel but has historically presented significantly different ABIs. To 
contain complexity and manage
maintainability, the Linux kernel community tries to prevent 
proliferation of unnecessarily vendor
specific ABIs. Often the opportunity for commonality is unclear until 
multiple vendors have proposed
separate interfaces [0].

The CCC as the home of open collaboration for Confidential Computing 
hardware vendors, software
vendors, and end users facilitates a unique environment for exchanging 
views on features and
requirements. Up to this point in time though most CCC collaboration has 
been on user space software
and use cases. The Linux kernel community has not been a regular part of 
discussions and the CCC has
not been a regular participant on the Linux Kernel Mail List (LKML)[1]. 
Bringing these communities
together in a structured forum has the promise of accelerating their 
joint work towards broad
availability of Confidential Computing in Linux environments.


# Scope:

1. Facilitate dialog between Linux kernel and Confidential Computing 
subject matter experts:
     * to facilitate direction for topics that need formal collaboration,
     * to have an additional venue to facilitate direction for topics 
that are stalled on LKML,
       which would benefit from higher bandwidth communication,
     * to have a common place to record decisions and formalize the 
output for others to reference,
     * and to introduce new technical topics emerging in either domain, 
e.g., attestation mechanisms
       approaching standardization.
2. Create recommendations for other organizations and communities such 
as LKML, IETF, etc.
    on topics at the intersection of the Linux kernel and Confidential 
Computing.


# Governance:

The CCC adheres to a culture of Minimum Viable Governance [2].
We prefer not to create rules that inhibit the progress of substantive work.

All are welcome at all CCC community forums. This SIG is targeted to all 
Linux kernel maintainers
and Confidential Computing vendors, project contributors, and end users 
with a dependence on the
Linux kernel. Meetings will follow standard CCC practices of inclusivity 
and openness adhering to
the community code of conduct [3] and the Linux Foundation’s Antitrust 
policy [4].


## Decision Making:

This SIG will use consensus decision making [5] wherein not all 
participants must be in full
agreement but that there should not be a strong consensus against a 
proposal [6].

In most cases we expect that once the SIG reaches consensus on a kernel 
topic, contributors will
propose aligned patches to LKML without any explicit communication from 
the SIG or the CCC.

If a formal CCC recommendation to another organization, e.g., IETF, TCG, 
eBPF Foundation, etc. is
required, the SIG will rely on the CCC Technical Advisory Council (TAC) 
to ratify proposals from the
SIG.

The pipeline of activity is expected (but not required) to look like this:
1. Identify feature need on mailing list or informal forum.
2. Exchange community proposals in the SIG.
3. Achieve informal consensus in the SIG.

     a. Author patches and take them to the appropriate community, e.g., 
LKML.

     or

     b. Raise an agenda item for the CCC TAC to formally ratify a 
proposal and publish to the
        appropriate organization.


## Initial Logistics:

The community will set its own meeting cadence starting from an initial 
cadence of meeting twice per
month. The CCC will provide a Zoom channel. At the end of each meeting a 
new chair may volunteer to
organize the agenda for the next meeting. The chair role confers no 
other responsibility or
authority beyond collecting agenda requests from the community and 
enforcing minutes and good order.
In addition to video meetings, the community will use an existing 
Discord channel for ongoing
communication. No mail list is required at this time.



# References:
[0] 
https://lore.kernel.org/all/64876bf6c30e2_1433ac29415@dwillia2-xfh.jf.intel.com.notmuch/

[1] https://lore.kernel.org/linux-coco/

[2] Sarah Novotny, Stephen Walli, “Minimum Viable Governance”, Linux 
Foundation Open Source Summit Europe 2020, 
https://www.youtube.com/watch?v=C2-_MjzP-Rs

[3] 
https://github.com/confidential-computing/governance/blob/main/code-of-conduct.md

[4] https://www.linuxfoundation.org/legal/antitrust-policy

[5] https://en.wikipedia.org/wiki/Consensus_decision-making

[6] See Rust’s Final Comment Period (FCP) https://github.com/rust-lang/rfcs

Re: RFC: CCC Linux Kernel SIG

[Carlos Bilbao]

On 11/7/23 19:24, Dan Middleton wrote:
> Hi from the CCC!
> 
> Many of our companies also collaborate in the Confidential Computing 
> Consortium.
> There we are proposing a SIG and invite your feedback and involvement:
> 
> https://github.com/confidential-computing/governance/pull/196
> 
> The entirety of the proposal is also copied below.
> 
> Thanks,
> 
> Dan Middleton
> CCC TAC Chair
> 
> 
> # Proposal: CCC Linux Kernel SIG
> 
> This proposal seeks to accelerate Confidential Computing feature velocity 
> in the Linux kernel.
> By creating a Linux Kernel SIG in the CCC we can foster community dialog to 
> develop vendor neutral
> architectural approaches to new features and thereby reduce complexity in 
> associated kernel patches.
> 
> 
> # Background:
> 
> One of the technical limitations to the adoption rate for Confidential 
> Computing is the speed at
> which we as a community can add features to the Linux kernel. Each vendor 
> has similar needs of the
> kernel but has historically presented significantly different ABIs. To 
> contain complexity and manage
> maintainability, the Linux kernel community tries to prevent proliferation 
> of unnecessarily vendor
> specific ABIs. Often the opportunity for commonality is unclear until 
> multiple vendors have proposed
> separate interfaces [0].
> 
> The CCC as the home of open collaboration for Confidential Computing 
> hardware vendors, software
> vendors, and end users facilitates a unique environment for exchanging 
> views on features and
> requirements. Up to this point in time though most CCC collaboration has 
> been on user space software
> and use cases. The Linux kernel community has not been a regular part of 
> discussions and the CCC has
> not been a regular participant on the Linux Kernel Mail List (LKML)[1]. 
> Bringing these communities
> together in a structured forum has the promise of accelerating their joint 
> work towards broad
> availability of Confidential Computing in Linux environments.
> 
> 
> # Scope:
> 
> 1. Facilitate dialog between Linux kernel and Confidential Computing 
> subject matter experts:
>      * to facilitate direction for topics that need formal collaboration,
>      * to have an additional venue to facilitate direction for topics that 
> are stalled on LKML,
>        which would benefit from higher bandwidth communication,
>      * to have a common place to record decisions and formalize the output 
> for others to reference,
>      * and to introduce new technical topics emerging in either domain, 
> e.g., attestation mechanisms
>        approaching standardization.
> 2. Create recommendations for other organizations and communities such as 
> LKML, IETF, etc.
>     on topics at the intersection of the Linux kernel and Confidential 
> Computing.
> 
> 
> # Governance:
> 
> The CCC adheres to a culture of Minimum Viable Governance [2].
> We prefer not to create rules that inhibit the progress of substantive work.
> 
> All are welcome at all CCC community forums. This SIG is targeted to all 
> Linux kernel maintainers
> and Confidential Computing vendors, project contributors, and end users 
> with a dependence on the
> Linux kernel. Meetings will follow standard CCC practices of inclusivity 
> and openness adhering to
> the community code of conduct [3] and the Linux Foundation’s Antitrust 
> policy [4].
> 
> 
> ## Decision Making:
> 
> This SIG will use consensus decision making [5] wherein not all 
> participants must be in full
> agreement but that there should not be a strong consensus against a 
> proposal [6].
> 
> In most cases we expect that once the SIG reaches consensus on a kernel 
> topic, contributors will
> propose aligned patches to LKML without any explicit communication from the 
> SIG or the CCC.
> 
> If a formal CCC recommendation to another organization, e.g., IETF, TCG, 
> eBPF Foundation, etc. is
> required, the SIG will rely on the CCC Technical Advisory Council (TAC) to 
> ratify proposals from the
> SIG.
> 
> The pipeline of activity is expected (but not required) to look like this:
> 1. Identify feature need on mailing list or informal forum.
> 2. Exchange community proposals in the SIG.
> 3. Achieve informal consensus in the SIG.
> 
>      a. Author patches and take them to the appropriate community, e.g., LKML.
> 
>      or
> 
>      b. Raise an agenda item for the CCC TAC to formally ratify a proposal 
> and publish to the
>         appropriate organization.
> 
> 
> ## Initial Logistics:
> 
> The community will set its own meeting cadence starting from an initial 
> cadence of meeting twice per
> month. The CCC will provide a Zoom channel. At the end of each meeting a 
> new chair may volunteer to
> organize the agenda for the next meeting. The chair role confers no other 
> responsibility or
> authority beyond collecting agenda requests from the community and 
> enforcing minutes and good order.
> In addition to video meetings, the community will use an existing Discord 
> channel for ongoing
> communication. No mail list is required at this time.

May I ask what Discord channel you're referring to? I'm assuming it's
open access.

> 
> 
> 
> # References:
> [0] 
> https://lore.kernel.org/all/64876bf6c30e2_1433ac29415@dwillia2-xfh.jf.intel.com.notmuch/
> 
> [1] https://lore.kernel.org/linux-coco/
> 
> [2] Sarah Novotny, Stephen Walli, “Minimum Viable Governance”, Linux 
> Foundation Open Source Summit Europe 2020, 
> https://www.youtube.com/watch?v=C2-_MjzP-Rs
> 
> [3] 
> https://github.com/confidential-computing/governance/blob/main/code-of-conduct.md
> 
> [4] https://www.linuxfoundation.org/legal/antitrust-policy
> 
> [5] https://en.wikipedia.org/wiki/Consensus_decision-making
> 
> [6] See Rust’s Final Comment Period (FCP) https://github.com/rust-lang/rfcs
> 
> 

Thanks,
Carlos

Re: RFC: CCC Linux Kernel SIG

[James Bottomley]

On Tue, 2023-11-07 at 19:24 -0600, Dan Middleton wrote:
> One of the technical limitations to the adoption rate for
> Confidential Computing is the speed at which we as a community can
> add features to the Linux kernel. Each vendor has similar needs of
> the kernel but has historically presented significantly different
> ABIs. To  contain complexity and manage maintainability, the Linux
> kernel community tries to prevent proliferation of unnecessarily
> vendor specific ABIs. Often the opportunity for commonality is
> unclear until multiple vendors have proposed separate interfaces [0].

The kernel community would see this as a feature not a bug.  It's very
difficult to appreciate the technical ramifications until you see the
actual code so even though it seems like more work in the beginning it
produces better and faster outcomes in the end.

To give the counter point to your example: each CC implementation had
its own ioctl set.  The proposal unifies it through a configfs
interface, but it's still not clear that the unification covers all the
use cases.  This, though, can be worked through after the acceptance of
the configfs pull request.  Adding ioctls is fairly simple, so there
wasn't really much wasted work before agreement was reached on a
preliminary design for configfs which can now be worked on by everyone.

[...]
> The Linux kernel community has not been a regular part of 
> discussions and the CCC has not been a regular participant on the
> Linux Kernel Mail List (LKML)[1].

Isn't this the current key problem with an obvious solution: the
interested members of the CCC can simply join the lists.  You could
make them do a formalised readout back to the CCC for people who don't
want to be at the kernel list sharp end and establish an action plan
from there if things are perceived to be going wrong.

> The pipeline of activity is expected (but not required) to look like
> this:
> 1. Identify feature need on mailing list or informal forum.
> 2. Exchange community proposals in the SIG.
> 3. Achieve informal consensus in the SIG.
> 
>      a. Author patches and take them to the appropriate community,
> e.g., LKML.

This really isn't the correct workflow for the Kernel.  The kernel
likes to see sketches in code when arguing about ABI and design.

The process really already works like your 1,2,3 with the exception
that 1. begins on the kernel or KVM list.  Trying to achieve consensus
in some other industrial forum and then present the patches to the list
as the unified consensus hasn't really worked out well in the past
(NVMe still has the scars to prove this).

[...]
> In addition to video meetings, the community will use an existing 
> Discord channel for ongoing communication

Just on a personal note: I've had no end of trouble with Discord,
primarily around the verification requirements for my email address
preventing me from getting an account from the single gatekeeper, so it
would be last on my list of useful systems for open conversations.

James

Re: RFC: CCC Linux Kernel SIG

[Dan Middleton]

On 11/9/23 11:22 AM, Carlos Bilbao wrote:

> On 11/7/23 19:24, Dan Middleton wrote:
>> <snip>
>> In addition to video meetings, the community will use an existing 
>> Discord channel for ongoing
>> communication. No mail list is required at this time.
>
> May I ask what Discord channel you're referring to? I'm assuming it's
> open access.
>
https://discord.com/channels/1123803528377413762/1124090655707234334

I believe it is open.
All CCC community forums zoom, slack, mail list, code are open.
However, this is a pre-existing channel... see below.

On 11/9/23 12:38 PM, James Bottomley wrote:
> Just on a personal note: I've had no end of trouble with Discord,
> primarily around the verification requirements for my email address
> preventing me from getting an account from the single gatekeeper, so it
> would be last on my list of useful systems for open conversations.

The CCC typically uses slack.

This discord reference is meant to accommodate kernel contributors who are
already communicating there. We don't want to break anything that's working.
Instead looking to bridge these different parts of the confidential 
computing
community who aren't in communication right now.

Thanks for the feedback and questions so far!


Regards,

Dan Middleton

Re: RFC: CCC Linux Kernel SIG

[Dionna Amalie Glaze]

On Thu, Nov 9, 2023 at 1:06 PM Dan Middleton
<dan.middleton@linux.intel.com> wrote:
>
> On 11/9/23 11:22 AM, Carlos Bilbao wrote:
>
> > On 11/7/23 19:24, Dan Middleton wrote:
> >> <snip>
> >> In addition to video meetings, the community will use an existing
> >> Discord channel for ongoing
> >> communication. No mail list is required at this time.
> >
> > May I ask what Discord channel you're referring to? I'm assuming it's
> > open access.
> >
> https://discord.com/channels/1123803528377413762/1124090655707234334
>
> I believe it is open.

This link does not work for me. "You find yourself in a strange place.
You don't have access to any text channels, or there are none in this
server."


-- 
-Dionna Glaze, PhD (she/her)

Re: RFC: CCC Linux Kernel SIG

[James Bottomley]

On Thu, 2023-11-09 at 15:05 -0600, Dan Middleton wrote:
> On 11/9/23 11:22 AM, Carlos Bilbao wrote:
> 
> > On 11/7/23 19:24, Dan Middleton wrote:
> > > <snip>
> > > In addition to video meetings, the community will use an existing
> > > Discord channel for ongoing
> > > communication. No mail list is required at this time.
> > 
> > May I ask what Discord channel you're referring to? I'm assuming
> > it's
> > open access.
> > 
> https://discord.com/channels/1123803528377413762/1124090655707234334
> 
> I believe it is open.
> All CCC community forums zoom, slack, mail list, code are open.
> However, this is a pre-existing channel... see below.
> 
> On 11/9/23 12:38 PM, James Bottomley wrote:
> > Just on a personal note: I've had no end of trouble with Discord,
> > primarily around the verification requirements for my email address
> > preventing me from getting an account from the single gatekeeper,
> > so it
> > would be last on my list of useful systems for open conversations.
> 
> The CCC typically uses slack.

For slack like but open chat, most communities are coalescing around
matrix (it's what Linux Plumbers will be using).  The kernel community
does still like IRC, though.


> This discord reference is meant to accommodate kernel contributors
> who are already communicating there.

OK, I'm not actually aware of any kernel conversations going on in
Discord (but then the kernel is a huge community, so I'm not aware of a
lot of things ...).
 
>  We don't want to break anything that's working.
> Instead looking to bridge these different parts of the confidential 
> computing community who aren't in communication right now.

As I said in my reply: I'd start on the existing Mailing lists. 
Designate someone at the CCC to watch and report and then relay any
feedback or concerns.  At least that starts to get you input without
having to have everyone in the CCC exposed to the kernel community.

James

Re: RFC: CCC Linux Kernel SIG

[Dan Middleton]

On 11/9/23 12:38 PM, James Bottomley wrote:
> On Tue, 2023-11-07 at 19:24 -0600, Dan Middleton wrote:
>> One of the technical limitations to the adoption rate for
>> Confidential Computing is the speed at which we as a community can
>> add features to the Linux kernel. Each vendor has similar needs of
>> the kernel but has historically presented significantly different
>> ABIs. To  contain complexity and manage maintainability, the Linux
>> kernel community tries to prevent proliferation of unnecessarily
>> vendor specific ABIs. Often the opportunity for commonality is
>> unclear until multiple vendors have proposed separate interfaces [0].
> The kernel community would see this as a feature not a bug.  It's very
> difficult to appreciate the technical ramifications until you see the
> actual code so even though it seems like more work in the beginning it
> produces better and faster outcomes in the end.
>
> To give the counter point to your example: each CC implementation had
> its own ioctl set.  The proposal unifies it through a configfs
> interface, but it's still not clear that the unification covers all the
> use cases.  This, though, can be worked through after the acceptance of
> the configfs pull request.  Adding ioctls is fairly simple, so there
> wasn't really much wasted work before agreement was reached on a
> preliminary design for configfs which can now be worked on by everyone.
>
> [...]

I agree we got there, but the path was quite long.
This might be the point where the second vendor ioctl was proposed which
would eventually give rise to the observation that a profusion of ioctl's
is bad:
https://lore.kernel.org/all/cover.1624719668.git.sathyanarayanan.kuppuswamy@linux.intel.com/

That was June 2021. It's now November 2023.

>> The Linux kernel community has not been a regular part of
>> discussions and the CCC has not been a regular participant on the
>> Linux Kernel Mail List (LKML)[1].
> Isn't this the current key problem with an obvious solution: the
> interested members of the CCC can simply join the lists.  You could
> make them do a formalised readout back to the CCC for people who don't
> want to be at the kernel list sharp end and establish an action plan
> from there if things are perceived to be going wrong.

I think that's a great suggestion.
That probably catches things already on the list.

What about getting in front of things before that stage?

>> The pipeline of activity is expected (but not required) to look like
>> this:
>> 1. Identify feature need on mailing list or informal forum.
>> 2. Exchange community proposals in the SIG.
>> 3. Achieve informal consensus in the SIG.
>>
>>       a. Author patches and take them to the appropriate community,
>> e.g., LKML.
> This really isn't the correct workflow for the Kernel.  The kernel
> likes to see sketches in code when arguing about ABI and design.
>
> The process really already works like your 1,2,3 with the exception
> that 1. begins on the kernel or KVM list.  Trying to achieve consensus
> in some other industrial forum and then present the patches to the list
> as the unified consensus hasn't really worked out well in the past
> (NVMe still has the scars to prove this).
>
> [...]
That's really interesting. I'd love any references about what didn't work
and what did work.

I think we could revise step 1 to read like.. learn about upcoming
features in the SIG and then recognize whether those will have kernel
impacts.

For example, it looks like most CPU vendors will be offering different
analogs of runtime measurement registers.

TEE-IO is coming down the pipe from CPU and device vendors.

There are software vendors working on novel use cases which might
imply different access patterns.

Would the CCC facilitating a mechanism to generate these cross-vendor
conversations be helpful?

What else would be helpful?

Thanks,
Dan Middleton