[LSF/MM/BPF BoF] Session for CXL memory

[LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

Hello,

I believe CXL memory is hot topic now. I believe we have multiple topics
for discussion. I personally would like to discuss CXL Fabric Manager
and vision of FM architecture implementation. I am going to share the topic
in separate email. I would like to suggest a special session for CXL memory
related topics.

How everybody feels about it?

Thanks,
Slava.
 

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

CC: LSF/MM/BPF mailing list. Sorry, missed the list.

> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
> 
> Hello,
> 
> I believe CXL memory is hot topic now. I believe we have multiple topics
> for discussion. I personally would like to discuss CXL Fabric Manager
> and vision of FM architecture implementation. I am going to share the topic
> in separate email. I would like to suggest a special session for CXL memory
> related topics.
> 
> How everybody feels about it?
> 
> Thanks,
> Slava.
> 

Re: [LSF/MM/BPF BoF] Session for CXL memory

[David Rientjes]

On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:

> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
> 
> > On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
> > 
> > Hello,
> > 
> > I believe CXL memory is hot topic now. I believe we have multiple topics
> > for discussion. I personally would like to discuss CXL Fabric Manager
> > and vision of FM architecture implementation. I am going to share the topic
> > in separate email. I would like to suggest a special session for CXL memory
> > related topics.
> > 
> > How everybody feels about it?
> > 

I think this makes a lot of sense, thanks for suggesting it.

Should this be a BoF or just a normal topic proposal?  I assume that there 
could be several different topics of interest all related to CXL.mem.

Specifically interesting would be the division of work between the kernel 
and userspace to manage memory placement on systems with locally attached 
CXL.  And, further, what APIs userspace would have at its disposal for 
explicit optimization of this placement that would exist beyond what is 
available for NUMA.

I assume we might also want to chat about CXL 2.0 extensions that would be 
useful, especially for cloud providers.

I'd be happy to join in any of the proposals for these topics, just let me 
know!

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Davidlohr Bueso]

On Sun, 22 Jan 2023, David Rientjes wrote:

>On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>
>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>>
>> > On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>> >
>> > Hello,
>> >
>> > I believe CXL memory is hot topic now. I believe we have multiple topics
>> > for discussion. I personally would like to discuss CXL Fabric Manager
>> > and vision of FM architecture implementation. I am going to share the topic
>> > in separate email. I would like to suggest a special session for CXL memory
>> > related topics.
>> >
>> > How everybody feels about it?
>> >
>
>I think this makes a lot of sense, thanks for suggesting it.

Yeah, considering all the CXL interest in last year's lsfmm, I think it would
be good to have a session.

>Should this be a BoF or just a normal topic proposal?  I assume that there
>could be several different topics of interest all related to CXL.mem.

fwiw last year there were 4-5 talks that directly touched on CXL (like half
day).

>Specifically interesting would be the division of work between the kernel
>and userspace to manage memory placement on systems with locally attached
>CXL.  And, further, what APIs userspace would have at its disposal for
>explicit optimization of this placement that would exist beyond what is
>available for NUMA.

+1

>
>I assume we might also want to chat about CXL 2.0 extensions that would be
>useful, especially for cloud providers.
>
>I'd be happy to join in any of the proposals for these topics, just let me
>know!

Likewise.

Thanks,
Davidlohr

Re: [External] [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

> On Jan 22, 2023, at 9:51 PM, David Rientjes <rientjes@google.com> wrote:
> 
> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
> 
>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>> 
>>> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>>> 
>>> Hello,
>>> 
>>> I believe CXL memory is hot topic now. I believe we have multiple topics
>>> for discussion. I personally would like to discuss CXL Fabric Manager
>>> and vision of FM architecture implementation. I am going to share the topic
>>> in separate email. I would like to suggest a special session for CXL memory
>>> related topics.
>>> 
>>> How everybody feels about it?
>>> 
> 
> I think this makes a lot of sense, thanks for suggesting it.
> 
> Should this be a BoF or just a normal topic proposal?  I assume that there 
> could be several different topics of interest all related to CXL.mem.
> 

My point is to have a dedicated CXL session where we can discuss
CXL related topics. And we can have likewise session if several CXL
related topics will be suggested. :)

> Specifically interesting would be the division of work between the kernel 
> and userspace to manage memory placement on systems with locally attached 
> CXL.  And, further, what APIs userspace would have at its disposal for 
> explicit optimization of this placement that would exist beyond what is 
> available for NUMA.
> 

Yes, I think this could be important discussion. Because, for example, I am working
on figuring out how functionality can be distributed among user-space, kernel-space,
and firmware for the case of Fabric Manager. And I believe it could be a good topic
that I have in mind.

> I assume we might also want to chat about CXL 2.0 extensions that would be 
> useful, especially for cloud providers.
> 

What CXL 2.0 extensions would be useful for cloud providers from your point of view?

Thanks,
Slava.

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Matthew Wilcox]

On Mon, Jan 23, 2023 at 05:46:00PM +0000, Adam Manzanares wrote:
> The session about DCDs at Plumbers comes to mind here. In addition, I think this

What's a DCD?

Re: [External] [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

> On Jan 23, 2023, at 10:29 AM, Matthew Wilcox <willy@infradead.org> wrote:
> 
> On Mon, Jan 23, 2023 at 05:46:00PM +0000, Adam Manzanares wrote:
>> The session about DCDs at Plumbers comes to mind here. In addition, I think this
> 
> What's a DCD?

DCD - Dynamic Capacity Device (CXL Specification 3.0)

Re: [External] [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

> On Jan 23, 2023, at 9:46 AM, Adam Manzanares <a.manzanares@samsung.com> wrote:
> 
> On Sun, Jan 22, 2023 at 09:51:19PM -0800, David Rientjes wrote:
>> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>> 
>>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>>> 
>>>> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>>>> 
>>>> Hello,
>>>> 
>>>> I believe CXL memory is hot topic now. I believe we have multiple topics
>>>> for discussion. I personally would like to discuss CXL Fabric Manager
>>>> and vision of FM architecture implementation. I am going to share the topic
>>>> in separate email. I would like to suggest a special session for CXL memory
>>>> related topics.
>>>> 
>>>> How everybody feels about it?
>>>> 
>> 
>> I think this makes a lot of sense, thanks for suggesting it.
>> 
>> Should this be a BoF or just a normal topic proposal?  I assume that there 
>> could be several different topics of interest all related to CXL.mem.
>> 
> 
> +1 for a normal topic proposal.

By the way, Samsung SMDK (memory development kit) suggested memory model
in user-space and kernel memory subsystem modification. I assume that we need
to discuss the memory model and kernel-space modification. So, it could be one of
the CXL related topic. Could we expect that Samsung guys deliver a talk?

Thanks,
Slava.

Re: [External] [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

> On Jan 26, 2023, at 8:58 AM, Adam Manzanares <a.manzanares@samsung.com> wrote:
> 
> On Mon, Jan 23, 2023 at 10:30:34AM -0800, Viacheslav A.Dubeyko wrote:
>> 
>> 
>>> On Jan 23, 2023, at 9:46 AM, Adam Manzanares <a.manzanares@samsung.com> wrote:
>>> 
>>> On Sun, Jan 22, 2023 at 09:51:19PM -0800, David Rientjes wrote:
>>>> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>>>> 
>>>>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>>>>> 
>>>>>> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> I believe CXL memory is hot topic now. I believe we have multiple topics
>>>>>> for discussion. I personally would like to discuss CXL Fabric Manager
>>>>>> and vision of FM architecture implementation. I am going to share the topic
>>>>>> in separate email. I would like to suggest a special session for CXL memory
>>>>>> related topics.
>>>>>> 
>>>>>> How everybody feels about it?
>>>>>> 
>>>> 
>>>> I think this makes a lot of sense, thanks for suggesting it.
>>>> 
>>>> Should this be a BoF or just a normal topic proposal?  I assume that there 
>>>> could be several different topics of interest all related to CXL.mem.
>>>> 
>>> 
>>> +1 for a normal topic proposal.
>> 
>> By the way, Samsung SMDK (memory development kit) suggested memory model
>> in user-space and kernel memory subsystem modification. I assume that we need
>> to discuss the memory model and kernel-space modification. So, it could be one of
>> the CXL related topic. Could we expect that Samsung guys deliver a talk?
> 
> +Kyungsan Kim
> 
> It is my understanding that Kyungsan, who is the lead for the SMDK project is 
> interested in sharing with the community some lessons learned from working with 
> CXL attached memory and a CFP is planned. 
> 

Sounds pretty great for me. :)

Thanks,
Slava.

Re: [External] [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

Hi Kyungsan,

> On Jan 28, 2023, at 5:45 PM, MTK <kim1158@gmail.com> wrote:
> 
> Thank you Mr. Slava and Adam for introducing our research.
> 
> If I was given an opportunity, I would like to share and discuss 
> - A case of memory tiering solution for CXL memory, Samsung SMDK, which works across userspace and kernelspace : why and how
> - Some lessens learned from Industry while CXL integration
> - Proposal for possible Linux MM expansions for CXL memory
> 
> Regarding FM topic, I'm also interested in what we should supports for CXL 3.0 and later topology.
> 

Please, send the topic suggestion then. :) You can check LSF/MM/BPF CFP page for guidance.

Thanks,
Slava.

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Yang Shi]

On Sun, Jan 22, 2023 at 9:51 PM David Rientjes <rientjes@google.com> wrote:
>
> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>
> > CC: LSF/MM/BPF mailing list. Sorry, missed the list.
> >
> > > On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
> > >
> > > Hello,
> > >
> > > I believe CXL memory is hot topic now. I believe we have multiple topics
> > > for discussion. I personally would like to discuss CXL Fabric Manager
> > > and vision of FM architecture implementation. I am going to share the topic
> > > in separate email. I would like to suggest a special session for CXL memory
> > > related topics.
> > >
> > > How everybody feels about it?
> > >
>
> I think this makes a lot of sense, thanks for suggesting it.
>
> Should this be a BoF or just a normal topic proposal?  I assume that there
> could be several different topics of interest all related to CXL.mem.
>
> Specifically interesting would be the division of work between the kernel
> and userspace to manage memory placement on systems with locally attached
> CXL.  And, further, what APIs userspace would have at its disposal for
> explicit optimization of this placement that would exist beyond what is
> available for NUMA.

Yeah, in addition, how CXL hierarchy interacts with memory tiering may
be an interesting topic too.

>
> I assume we might also want to chat about CXL 2.0 extensions that would be
> useful, especially for cloud providers.
>
> I'd be happy to join in any of the proposals for these topics, just let me
> know!

+1 for me.

>

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Wei Xu]

On Mon, Jan 23, 2023 at 4:22 PM Yang Shi <shy828301@gmail.com> wrote:
>
> On Sun, Jan 22, 2023 at 9:51 PM David Rientjes <rientjes@google.com> wrote:
> >
> > On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
> >
> > > CC: LSF/MM/BPF mailing list. Sorry, missed the list.
> > >
> > > > On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I believe CXL memory is hot topic now. I believe we have multiple topics
> > > > for discussion. I personally would like to discuss CXL Fabric Manager
> > > > and vision of FM architecture implementation. I am going to share the topic
> > > > in separate email. I would like to suggest a special session for CXL memory
> > > > related topics.
> > > >
> > > > How everybody feels about it?
> > > >
> >
> > I think this makes a lot of sense, thanks for suggesting it.
> >
> > Should this be a BoF or just a normal topic proposal?  I assume that there
> > could be several different topics of interest all related to CXL.mem.
> >
> > Specifically interesting would be the division of work between the kernel
> > and userspace to manage memory placement on systems with locally attached
> > CXL.  And, further, what APIs userspace would have at its disposal for
> > explicit optimization of this placement that would exist beyond what is
> > available for NUMA.
>
> Yeah, in addition, how CXL hierarchy interacts with memory tiering may
> be an interesting topic too.
>
> >
> > I assume we might also want to chat about CXL 2.0 extensions that would be
> > useful, especially for cloud providers.
> >
> > I'd be happy to join in any of the proposals for these topics, just let me
> > know!
>
> +1 for me.
>

I am also interested in these discussions.

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Zhu Yanjun]

在 2023/1/24 8:57, Wei Xu 写道:
> On Mon, Jan 23, 2023 at 4:22 PM Yang Shi <shy828301@gmail.com> wrote:
>>
>> On Sun, Jan 22, 2023 at 9:51 PM David Rientjes <rientjes@google.com> wrote:
>>>
>>> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>>>
>>>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>>>>
>>>>> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I believe CXL memory is hot topic now. I believe we have multiple topics
>>>>> for discussion. I personally would like to discuss CXL Fabric Manager
>>>>> and vision of FM architecture implementation. I am going to share the topic
>>>>> in separate email. I would like to suggest a special session for CXL memory
>>>>> related topics.
>>>>>
>>>>> How everybody feels about it?
>>>>>
>>>
>>> I think this makes a lot of sense, thanks for suggesting it.
>>>
>>> Should this be a BoF or just a normal topic proposal?  I assume that there
>>> could be several different topics of interest all related to CXL.mem.
>>>
>>> Specifically interesting would be the division of work between the kernel
>>> and userspace to manage memory placement on systems with locally attached
>>> CXL.  And, further, what APIs userspace would have at its disposal for
>>> explicit optimization of this placement that would exist beyond what is
>>> available for NUMA.
>>
>> Yeah, in addition, how CXL hierarchy interacts with memory tiering may
>> be an interesting topic too.
>>
>>>
>>> I assume we might also want to chat about CXL 2.0 extensions that would be
>>> useful, especially for cloud providers.
>>>
>>> I'd be happy to join in any of the proposals for these topics, just let me
>>> know!
>>
>> +1 for me.
>>
> 
> I am also interested in these discussions.

+1， I am interested in this.

> 

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Dragan Stancevic]

On 1/25/23 09:04, Zhu Yanjun wrote:
> 在 2023/1/24 8:57, Wei Xu 写道:
>> On Mon, Jan 23, 2023 at 4:22 PM Yang Shi <shy828301@gmail.com> wrote:
>>>
>>> On Sun, Jan 22, 2023 at 9:51 PM David Rientjes <rientjes@google.com> 
>>> wrote:
>>>>
>>>> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>>>>
>>>>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>>>>>
>>>>>> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko 
>>>>>> <viacheslav.dubeyko@bytedance.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I believe CXL memory is hot topic now. I believe we have multiple 
>>>>>> topics
>>>>>> for discussion. I personally would like to discuss CXL Fabric Manager
>>>>>> and vision of FM architecture implementation. I am going to share 
>>>>>> the topic
>>>>>> in separate email. I would like to suggest a special session for 
>>>>>> CXL memory
>>>>>> related topics.
>>>>>>
>>>>>> How everybody feels about it?
>>>>>>
>>>>
>>>> I think this makes a lot of sense, thanks for suggesting it.
>>>>
>>>> Should this be a BoF or just a normal topic proposal?  I assume that 
>>>> there
>>>> could be several different topics of interest all related to CXL.mem.
>>>>
>>>> Specifically interesting would be the division of work between the 
>>>> kernel
>>>> and userspace to manage memory placement on systems with locally 
>>>> attached
>>>> CXL.  And, further, what APIs userspace would have at its disposal for
>>>> explicit optimization of this placement that would exist beyond what is
>>>> available for NUMA.
>>>
>>> Yeah, in addition, how CXL hierarchy interacts with memory tiering may
>>> be an interesting topic too.
>>>
>>>>
>>>> I assume we might also want to chat about CXL 2.0 extensions that 
>>>> would be
>>>> useful, especially for cloud providers.
>>>>
>>>> I'd be happy to join in any of the proposals for these topics, just 
>>>> let me
>>>> know!
>>>
>>> +1 for me.
>>>
>>
>> I am also interested in these discussions.
> 
> +1， I am interested in this.

+1, also interested in cloud aspects


--
Peace can only come as a natural consequence
of universal enlightenment -Dr. Nikola Tesla

Re: [LSF/MM/BPF BoF] Session for CXL memory

[Aneesh Kumar K.V]

Yang Shi <shy828301@gmail.com> writes:

> On Sun, Jan 22, 2023 at 9:51 PM David Rientjes <rientjes@google.com> wrote:
>>
>> On Fri, 6 Jan 2023, Viacheslav A.Dubeyko wrote:
>>
>> > CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>> >
>> > > On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>> > >
>> > > Hello,
>> > >
>> > > I believe CXL memory is hot topic now. I believe we have multiple topics
>> > > for discussion. I personally would like to discuss CXL Fabric Manager
>> > > and vision of FM architecture implementation. I am going to share the topic
>> > > in separate email. I would like to suggest a special session for CXL memory
>> > > related topics.
>> > >
>> > > How everybody feels about it?
>> > >
>>
>> I think this makes a lot of sense, thanks for suggesting it.
>>
>> Should this be a BoF or just a normal topic proposal?  I assume that there
>> could be several different topics of interest all related to CXL.mem.
>>
>> Specifically interesting would be the division of work between the kernel
>> and userspace to manage memory placement on systems with locally attached
>> CXL.  And, further, what APIs userspace would have at its disposal for
>> explicit optimization of this placement that would exist beyond what is
>> available for NUMA.
>
> Yeah, in addition, how CXL hierarchy interacts with memory tiering may
> be an interesting topic too.
>
>>
>> I assume we might also want to chat about CXL 2.0 extensions that would be
>> useful, especially for cloud providers.
>>
>> I'd be happy to join in any of the proposals for these topics, just let me
>> know!
>
> +1 for me.
>

I would also be interested in finding out what we learned about the
device attributes that people want to use for building hierarchy. We
still haven't hooked up HMAT/CDAT to memory tiers. It would be good to
understand and discuss on how we should make progress here.

-aneesh

Re: [External] [LSF/MM/BPF BoF] Session for CXL memory

[Viacheslav A.Dubeyko]

> On Jan 24, 2023, at 2:12 AM, Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:
> 
> On Fri, 6 Jan 2023 14:20:42 -0800
> "Viacheslav A.Dubeyko" <viacheslav.dubeyko@bytedance.com> wrote:
> 
>> CC: LSF/MM/BPF mailing list. Sorry, missed the list.
>> 
>>> On Jan 6, 2023, at 11:51 AM, Viacheslav A.Dubeyko <viacheslav.dubeyko@bytedance.com> wrote:
>>> 
>>> Hello,
>>> 
>>> I believe CXL memory is hot topic now. I believe we have multiple topics
>>> for discussion. I personally would like to discuss CXL Fabric Manager
>>> and vision of FM architecture implementation. 
> 
> The fabric manager is rather disconnected from the host side of things (all
> out of band communications), so it would be a stretch to build a stand alone
> topic around that for LSF-MM. If we have the right people in the room /
> online, it would be good to discuss it (so part of a wider sessions on CXL).
> I'd love to get some traction before LSFMM though! Host aspects such as
> Dynamic Capacity Devices and sharing feel more LSFMM suitable.
> 
>>> I am going to share the topic in separate email.
> 
> Did I miss the email, or not sent yet?  That topic is obscure enough we definitely
> need some background if anyone outside of CXL folk is going to have any idea what
> we are talking about.
> 

You missed nothing. :) I am still polishing the FM related topic. Sorry, I was busy
with other tasks. 

>>> would like to suggest a special session for CXL memory
>>> related topics.
>>> 
>>> How everybody feels about it?
> 
> A lot of the interesting bits currently strike me as rather speculative
> (no code), so sessions might not be as productive as shooting at an
> implementation. That's less true of FM stuff as we really do need
> an outline of an architecture plus some planning on that.
> 
> Could we have something to shoot at for other topics in the
> time frame?  maybe...
> 

I believe even discussion before implementation could make sense.
Because, it provides the way to make the architecture/API vision more
clear and understandable by everyone sometimes. :)

Thanks,
Slava.