CXL volatile memory is not listed

CXL volatile memory is not listed

[Maverickk 78]

Hello,

I am running qemu-system-x86_64

qemu-system-x86_64 --version
QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)

qemu-system-x86_64 \
-m 2G,slots=4,maxmem=4G \
-smp 4 \
-machine type=q35,accel=kvm,cxl=on \
-enable-kvm \
-nographic \
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
-object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
-device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G


I was expecting the CXL memory to be listed in "System Ram", the lsmem
shows only 2G memory which is System RAM, it's not listing the CXL
memory.

Do I need to pass any particular parameter in the kernel command line?

Is there any documentation available? I followed the inputs provided in

https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/

Is there any documentation/blog listed?

Re: CXL volatile memory is not listed

[Jonathan Cameron via]

On Wed, 9 Aug 2023 04:21:47 +0530
Maverickk 78 <maverickk1778@gmail.com> wrote:

> Hello,
> 
> I am running qemu-system-x86_64
> 
> qemu-system-x86_64 --version
> QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
>
+Cc linux-cxl as the answer is more todo with linux than qemu.
 
> qemu-system-x86_64 \
> -m 2G,slots=4,maxmem=4G \
> -smp 4 \
> -machine type=q35,accel=kvm,cxl=on \
> -enable-kvm \
> -nographic \
> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G

There are some problems upstream at the moment (probably not cxl related but
I'm digging). So today I can't boot an x86 machine. (goody)


More generally for the flow that would bring the memory up as system ram
you would typically need the bios to have done the CXL enumeration or
a bunch of scripts in the kernel to have done it.  In general it can't
be fully automated, because there are policy decisions to make on things like
interleaving.

I'm not aware of any open source BIOSs that do it yet.  So you have
to rely on the same kernel paths as for persistent memory - manual configuration
etc in the kernel. 

There is support in ndctl for those enabling flows, so I'd look there
for more information

Jonathan


> 
> 
> I was expecting the CXL memory to be listed in "System Ram", the lsmem
> shows only 2G memory which is System RAM, it's not listing the CXL
> memory.
> 
> Do I need to pass any particular parameter in the kernel command line?
> 
> Is there any documentation available? I followed the inputs provided in
> 
> https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> 
> Is there any documentation/blog listed?

Re: CXL volatile memory is not listed

[Maverickk 78]

Jonathan,

> More generally for the flow that would bring the memory up as system ram
> you would typically need the bios to have done the CXL enumeration or
> a bunch of scripts in the kernel to have done it.  In general it can't
> be fully automated, because there are policy decisions to make on things like
> interleaving.

BIOS CXL enumeration? is CEDT not enough? or BIOS further needs to
create an entry
in the e820 table?

>
> I'm not aware of any open source BIOSs that do it yet.  So you have
> to rely on the same kernel paths as for persistent memory - manual configuration
> etc in the kernel.
>
Manual works with "cxl create regiton"  :)

On Thu, 10 Aug 2023 at 16:05, Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>
> On Wed, 9 Aug 2023 04:21:47 +0530
> Maverickk 78 <maverickk1778@gmail.com> wrote:
>
> > Hello,
> >
> > I am running qemu-system-x86_64
> >
> > qemu-system-x86_64 --version
> > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> >
> +Cc linux-cxl as the answer is more todo with linux than qemu.
>
> > qemu-system-x86_64 \
> > -m 2G,slots=4,maxmem=4G \
> > -smp 4 \
> > -machine type=q35,accel=kvm,cxl=on \
> > -enable-kvm \
> > -nographic \
> > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
>
> There are some problems upstream at the moment (probably not cxl related but
> I'm digging). So today I can't boot an x86 machine. (goody)
>
>
> More generally for the flow that would bring the memory up as system ram
> you would typically need the bios to have done the CXL enumeration or
> a bunch of scripts in the kernel to have done it.  In general it can't
> be fully automated, because there are policy decisions to make on things like
> interleaving.
>
> I'm not aware of any open source BIOSs that do it yet.  So you have
> to rely on the same kernel paths as for persistent memory - manual configuration
> etc in the kernel.
>
> There is support in ndctl for those enabling flows, so I'd look there
> for more information
>
> Jonathan
>
>
> >
> >
> > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > shows only 2G memory which is System RAM, it's not listing the CXL
> > memory.
> >
> > Do I need to pass any particular parameter in the kernel command line?
> >
> > Is there any documentation available? I followed the inputs provided in
> >
> > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> >
> > Is there any documentation/blog listed?
>

Re: CXL volatile memory is not listed

[Jonathan Cameron via]

On Fri, 11 Aug 2023 08:04:26 +0530
Maverickk 78 <maverickk1778@gmail.com> wrote:

> Jonathan,
> 
> > More generally for the flow that would bring the memory up as system ram
> > you would typically need the bios to have done the CXL enumeration or
> > a bunch of scripts in the kernel to have done it.  In general it can't
> > be fully automated, because there are policy decisions to make on things like
> > interleaving.  
> 
> BIOS CXL enumeration? is CEDT not enough? or BIOS further needs to
> create an entry
> in the e820 table?
On intel platforms 'maybe' :)  I know how it works on those that just
use the nice standard EFI tables - less familiar with the e820 stuff :)

CEDT says where to find the the various bits of system related CXL stuff.
Nothing in there on the configuration that should be used such as interleaving
as that depends on what the administrator wants. Or on what the BIOS has
decided the users should have.

> 
> >
> > I'm not aware of any open source BIOSs that do it yet.  So you have
> > to rely on the same kernel paths as for persistent memory - manual configuration
> > etc in the kernel.
> >  
> Manual works with "cxl create regiton"  :)
Great.

Jonathan

> 
> On Thu, 10 Aug 2023 at 16:05, Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> >
> > On Wed, 9 Aug 2023 04:21:47 +0530
> > Maverickk 78 <maverickk1778@gmail.com> wrote:
> >  
> > > Hello,
> > >
> > > I am running qemu-system-x86_64
> > >
> > > qemu-system-x86_64 --version
> > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > >  
> > +Cc linux-cxl as the answer is more todo with linux than qemu.
> >  
> > > qemu-system-x86_64 \
> > > -m 2G,slots=4,maxmem=4G \
> > > -smp 4 \
> > > -machine type=q35,accel=kvm,cxl=on \
> > > -enable-kvm \
> > > -nographic \
> > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G  
> >
> > There are some problems upstream at the moment (probably not cxl related but
> > I'm digging). So today I can't boot an x86 machine. (goody)
> >
> >
> > More generally for the flow that would bring the memory up as system ram
> > you would typically need the bios to have done the CXL enumeration or
> > a bunch of scripts in the kernel to have done it.  In general it can't
> > be fully automated, because there are policy decisions to make on things like
> > interleaving.
> >
> > I'm not aware of any open source BIOSs that do it yet.  So you have
> > to rely on the same kernel paths as for persistent memory - manual configuration
> > etc in the kernel.
> >
> > There is support in ndctl for those enabling flows, so I'd look there
> > for more information
> >
> > Jonathan
> >
> >  
> > >
> > >
> > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > memory.
> > >
> > > Do I need to pass any particular parameter in the kernel command line?
> > >
> > > Is there any documentation available? I followed the inputs provided in
> > >
> > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > >
> > > Is there any documentation/blog listed?  
> >  

Re: CXL volatile memory is not listed

[Maverickk 78]

Hi Jonathan,

The use case of CXL switch will always need some sort of management
agent + FM configuring the available CXL memory connected.

In most cases it would be bmc controller configuring MLD/MHD's to
host, and in very rare scenarios it may be one of the host interacting
with FM firmware inside the switch which would do the trick.

Another use case is static hardcoding between CXL memory & host in
built in cxl switch

There is no scenario where one of the host BIOS can push the select
CXL memory to itself.


Is my understanding correct?



On Fri, 11 Aug 2023 at 19:25, Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>
> On Fri, 11 Aug 2023 08:04:26 +0530
> Maverickk 78 <maverickk1778@gmail.com> wrote:
>
> > Jonathan,
> >
> > > More generally for the flow that would bring the memory up as system ram
> > > you would typically need the bios to have done the CXL enumeration or
> > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > be fully automated, because there are policy decisions to make on things like
> > > interleaving.
> >
> > BIOS CXL enumeration? is CEDT not enough? or BIOS further needs to
> > create an entry
> > in the e820 table?
> On intel platforms 'maybe' :)  I know how it works on those that just
> use the nice standard EFI tables - less familiar with the e820 stuff :)
>
> CEDT says where to find the the various bits of system related CXL stuff.
> Nothing in there on the configuration that should be used such as interleaving
> as that depends on what the administrator wants. Or on what the BIOS has
> decided the users should have.
>
> >
> > >
> > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > etc in the kernel.
> > >
> > Manual works with "cxl create regiton"  :)
> Great.
>
> Jonathan
>
> >
> > On Thu, 10 Aug 2023 at 16:05, Jonathan Cameron
> > <Jonathan.Cameron@huawei.com> wrote:
> > >
> > > On Wed, 9 Aug 2023 04:21:47 +0530
> > > Maverickk 78 <maverickk1778@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I am running qemu-system-x86_64
> > > >
> > > > qemu-system-x86_64 --version
> > > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > > >
> > > +Cc linux-cxl as the answer is more todo with linux than qemu.
> > >
> > > > qemu-system-x86_64 \
> > > > -m 2G,slots=4,maxmem=4G \
> > > > -smp 4 \
> > > > -machine type=q35,accel=kvm,cxl=on \
> > > > -enable-kvm \
> > > > -nographic \
> > > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> > >
> > > There are some problems upstream at the moment (probably not cxl related but
> > > I'm digging). So today I can't boot an x86 machine. (goody)
> > >
> > >
> > > More generally for the flow that would bring the memory up as system ram
> > > you would typically need the bios to have done the CXL enumeration or
> > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > be fully automated, because there are policy decisions to make on things like
> > > interleaving.
> > >
> > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > etc in the kernel.
> > >
> > > There is support in ndctl for those enabling flows, so I'd look there
> > > for more information
> > >
> > > Jonathan
> > >
> > >
> > > >
> > > >
> > > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > > memory.
> > > >
> > > > Do I need to pass any particular parameter in the kernel command line?
> > > >
> > > > Is there any documentation available? I followed the inputs provided in
> > > >
> > > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > > >
> > > > Is there any documentation/blog listed?
> > >
>

Re: CXL volatile memory is not listed

[Shreyas Shah via]

[-- Attachment #1: Type: text/plain, Size: 5137 bytes --]

Once the cxl memory is online, how does Operating system know whether to malloc in the cxl memory or socket attached DDR memory?



[https://static1.squarespace.com/static/60dbbd6d597c966b91a3b27b/t/6104415e6177af1589fb05e4/1627668830393/elastics-cloud-logo-120.png]<https://www.elastics.cloud/>
Shreyas Shah
Founder, CTO and Chief Scientist, Elastics.cloud, Inc.
1730 North First Street, 5th Floor, San Jose, CA 95112
t: 408 476 3100<tel:408%20476%203100> | e: email: shreyas.shah@elastics.cloud

________________________________
From: Maverickk 78 <maverickk1778@gmail.com>
Sent: Thursday, August 17, 2023 10:18 PM
To: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: Jonathan Cameron via <qemu-devel@nongnu.org>; linux-cxl@vger.kernel.org <linux-cxl@vger.kernel.org>
Subject: Re: CXL volatile memory is not listed

Hi Jonathan,

The use case of CXL switch will always need some sort of management
agent + FM configuring the available CXL memory connected.

In most cases it would be bmc controller configuring MLD/MHD's to
host, and in very rare scenarios it may be one of the host interacting
with FM firmware inside the switch which would do the trick.

Another use case is static hardcoding between CXL memory & host in
built in cxl switch

There is no scenario where one of the host BIOS can push the select
CXL memory to itself.


Is my understanding correct?



On Fri, 11 Aug 2023 at 19:25, Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>
> On Fri, 11 Aug 2023 08:04:26 +0530
> Maverickk 78 <maverickk1778@gmail.com> wrote:
>
> > Jonathan,
> >
> > > More generally for the flow that would bring the memory up as system ram
> > > you would typically need the bios to have done the CXL enumeration or
> > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > be fully automated, because there are policy decisions to make on things like
> > > interleaving.
> >
> > BIOS CXL enumeration? is CEDT not enough? or BIOS further needs to
> > create an entry
> > in the e820 table?
> On intel platforms 'maybe' :)  I know how it works on those that just
> use the nice standard EFI tables - less familiar with the e820 stuff :)
>
> CEDT says where to find the the various bits of system related CXL stuff.
> Nothing in there on the configuration that should be used such as interleaving
> as that depends on what the administrator wants. Or on what the BIOS has
> decided the users should have.
>
> >
> > >
> > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > etc in the kernel.
> > >
> > Manual works with "cxl create regiton"  :)
> Great.
>
> Jonathan
>
> >
> > On Thu, 10 Aug 2023 at 16:05, Jonathan Cameron
> > <Jonathan.Cameron@huawei.com> wrote:
> > >
> > > On Wed, 9 Aug 2023 04:21:47 +0530
> > > Maverickk 78 <maverickk1778@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I am running qemu-system-x86_64
> > > >
> > > > qemu-system-x86_64 --version
> > > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > > >
> > > +Cc linux-cxl as the answer is more todo with linux than qemu.
> > >
> > > > qemu-system-x86_64 \
> > > > -m 2G,slots=4,maxmem=4G \
> > > > -smp 4 \
> > > > -machine type=q35,accel=kvm,cxl=on \
> > > > -enable-kvm \
> > > > -nographic \
> > > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> > >
> > > There are some problems upstream at the moment (probably not cxl related but
> > > I'm digging). So today I can't boot an x86 machine. (goody)
> > >
> > >
> > > More generally for the flow that would bring the memory up as system ram
> > > you would typically need the bios to have done the CXL enumeration or
> > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > be fully automated, because there are policy decisions to make on things like
> > > interleaving.
> > >
> > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > etc in the kernel.
> > >
> > > There is support in ndctl for those enabling flows, so I'd look there
> > > for more information
> > >
> > > Jonathan
> > >
> > >
> > > >
> > > >
> > > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > > memory.
> > > >
> > > > Do I need to pass any particular parameter in the kernel command line?
> > > >
> > > > Is there any documentation available? I followed the inputs provided in
> > > >
> > > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > > >
> > > > Is there any documentation/blog listed?
> > >
>

[-- Attachment #2: Type: text/html, Size: 8810 bytes --]

Re: CXL volatile memory is not listed

[Jonathan Cameron via]

On Fri, 18 Aug 2023 11:30:55 +0000
Shreyas Shah <shreyas.shah@elastics.cloud> wrote:

> Once the cxl memory is online, how does Operating system know whether to malloc in the cxl memory or socket attached DDR memory?

If you've brought the memory up as 'normal memory' via kmem rather than the other dax options
then it'll be a separate NUMA node.  Hence you can control allocations using same tools
used on multiple numa node systems.

Jonathan

> 
> 
> 
> [https://static1.squarespace.com/static/60dbbd6d597c966b91a3b27b/t/6104415e6177af1589fb05e4/1627668830393/elastics-cloud-logo-120.png]<https://www.elastics.cloud/>
> Shreyas Shah
> Founder, CTO and Chief Scientist, Elastics.cloud, Inc.
> 1730 North First Street, 5th Floor, San Jose, CA 95112
> t: 408 476 3100<tel:408%20476%203100> | e: email: shreyas.shah@elastics.cloud
> 
> ________________________________
> From: Maverickk 78 <maverickk1778@gmail.com>
> Sent: Thursday, August 17, 2023 10:18 PM
> To: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Cc: Jonathan Cameron via <qemu-devel@nongnu.org>; linux-cxl@vger.kernel.org <linux-cxl@vger.kernel.org>
> Subject: Re: CXL volatile memory is not listed
> 
> Hi Jonathan,
> 
> The use case of CXL switch will always need some sort of management
> agent + FM configuring the available CXL memory connected.
> 
> In most cases it would be bmc controller configuring MLD/MHD's to
> host, and in very rare scenarios it may be one of the host interacting
> with FM firmware inside the switch which would do the trick.
> 
> Another use case is static hardcoding between CXL memory & host in
> built in cxl switch
> 
> There is no scenario where one of the host BIOS can push the select
> CXL memory to itself.
> 
> 
> Is my understanding correct?
> 
> 
> 
> On Fri, 11 Aug 2023 at 19:25, Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> >
> > On Fri, 11 Aug 2023 08:04:26 +0530
> > Maverickk 78 <maverickk1778@gmail.com> wrote:
> >  
> > > Jonathan,
> > >  
> > > > More generally for the flow that would bring the memory up as system ram
> > > > you would typically need the bios to have done the CXL enumeration or
> > > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > > be fully automated, because there are policy decisions to make on things like
> > > > interleaving.  
> > >
> > > BIOS CXL enumeration? is CEDT not enough? or BIOS further needs to
> > > create an entry
> > > in the e820 table?  
> > On intel platforms 'maybe' :)  I know how it works on those that just
> > use the nice standard EFI tables - less familiar with the e820 stuff :)
> >
> > CEDT says where to find the the various bits of system related CXL stuff.
> > Nothing in there on the configuration that should be used such as interleaving
> > as that depends on what the administrator wants. Or on what the BIOS has
> > decided the users should have.
> >  
> > >  
> > > >
> > > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > > etc in the kernel.
> > > >  
> > > Manual works with "cxl create regiton"  :)  
> > Great.
> >
> > Jonathan
> >  
> > >
> > > On Thu, 10 Aug 2023 at 16:05, Jonathan Cameron
> > > <Jonathan.Cameron@huawei.com> wrote:  
> > > >
> > > > On Wed, 9 Aug 2023 04:21:47 +0530
> > > > Maverickk 78 <maverickk1778@gmail.com> wrote:
> > > >  
> > > > > Hello,
> > > > >
> > > > > I am running qemu-system-x86_64
> > > > >
> > > > > qemu-system-x86_64 --version
> > > > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > > > >  
> > > > +Cc linux-cxl as the answer is more todo with linux than qemu.
> > > >  
> > > > > qemu-system-x86_64 \
> > > > > -m 2G,slots=4,maxmem=4G \
> > > > > -smp 4 \
> > > > > -machine type=q35,accel=kvm,cxl=on \
> > > > > -enable-kvm \
> > > > > -nographic \
> > > > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G  
> > > >
> > > > There are some problems upstream at the moment (probably not cxl related but
> > > > I'm digging). So today I can't boot an x86 machine. (goody)
> > > >
> > > >
> > > > More generally for the flow that would bring the memory up as system ram
> > > > you would typically need the bios to have done the CXL enumeration or
> > > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > > be fully automated, because there are policy decisions to make on things like
> > > > interleaving.
> > > >
> > > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > > etc in the kernel.
> > > >
> > > > There is support in ndctl for those enabling flows, so I'd look there
> > > > for more information
> > > >
> > > > Jonathan
> > > >
> > > >  
> > > > >
> > > > >
> > > > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > > > memory.
> > > > >
> > > > > Do I need to pass any particular parameter in the kernel command line?
> > > > >
> > > > > Is there any documentation available? I followed the inputs provided in
> > > > >
> > > > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > > > >
> > > > > Is there any documentation/blog listed?  
> > > >  
> >  
> 

Re: CXL volatile memory is not listed

[Jonathan Cameron via]

On Fri, 18 Aug 2023 10:48:35 +0530
Maverickk 78 <maverickk1778@gmail.com> wrote:

> Hi Jonathan,
> 
> The use case of CXL switch will always need some sort of management
> agent + FM configuring the available CXL memory connected.
> 
> In most cases it would be bmc controller configuring MLD/MHD's to
> host, and in very rare scenarios it may be one of the host interacting
> with FM firmware inside the switch which would do the trick.
> 
> Another use case is static hardcoding between CXL memory & host in
> built in cxl switch
> 
> There is no scenario where one of the host BIOS can push the select
> CXL memory to itself.
> 
> 
> Is my understanding correct?

It's possible that a particular set of systems work on the basis
of the FM (BMC) in a memory appliance etc having already set up the
memory before the hosts using it are booted.  This would be done for
legacy systems / unaware operating systems for instances.
In those cases the BIOS could enumerate and configure everything present
when it starts and provide that info the the OS running on the host as
EFI (and/or e820) and SRAT etc.

If doing more dynamic memory pooling, I'd expect the OS to do all the
hard work.  Note a common case in real systems is likely to be Multi Head
devices for memory pooling, but they also require configuration before the
memory is available to the host, so the points above are the same.

Jonathan

> 
> 
> 
> On Fri, 11 Aug 2023 at 19:25, Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> >
> > On Fri, 11 Aug 2023 08:04:26 +0530
> > Maverickk 78 <maverickk1778@gmail.com> wrote:
> >  
> > > Jonathan,
> > >  
> > > > More generally for the flow that would bring the memory up as system ram
> > > > you would typically need the bios to have done the CXL enumeration or
> > > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > > be fully automated, because there are policy decisions to make on things like
> > > > interleaving.  
> > >
> > > BIOS CXL enumeration? is CEDT not enough? or BIOS further needs to
> > > create an entry
> > > in the e820 table?  
> > On intel platforms 'maybe' :)  I know how it works on those that just
> > use the nice standard EFI tables - less familiar with the e820 stuff :)
> >
> > CEDT says where to find the the various bits of system related CXL stuff.
> > Nothing in there on the configuration that should be used such as interleaving
> > as that depends on what the administrator wants. Or on what the BIOS has
> > decided the users should have.
> >  
> > >  
> > > >
> > > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > > etc in the kernel.
> > > >  
> > > Manual works with "cxl create regiton"  :)  
> > Great.
> >
> > Jonathan
> >  
> > >
> > > On Thu, 10 Aug 2023 at 16:05, Jonathan Cameron
> > > <Jonathan.Cameron@huawei.com> wrote:  
> > > >
> > > > On Wed, 9 Aug 2023 04:21:47 +0530
> > > > Maverickk 78 <maverickk1778@gmail.com> wrote:
> > > >  
> > > > > Hello,
> > > > >
> > > > > I am running qemu-system-x86_64
> > > > >
> > > > > qemu-system-x86_64 --version
> > > > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > > > >  
> > > > +Cc linux-cxl as the answer is more todo with linux than qemu.
> > > >  
> > > > > qemu-system-x86_64 \
> > > > > -m 2G,slots=4,maxmem=4G \
> > > > > -smp 4 \
> > > > > -machine type=q35,accel=kvm,cxl=on \
> > > > > -enable-kvm \
> > > > > -nographic \
> > > > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G  
> > > >
> > > > There are some problems upstream at the moment (probably not cxl related but
> > > > I'm digging). So today I can't boot an x86 machine. (goody)
> > > >
> > > >
> > > > More generally for the flow that would bring the memory up as system ram
> > > > you would typically need the bios to have done the CXL enumeration or
> > > > a bunch of scripts in the kernel to have done it.  In general it can't
> > > > be fully automated, because there are policy decisions to make on things like
> > > > interleaving.
> > > >
> > > > I'm not aware of any open source BIOSs that do it yet.  So you have
> > > > to rely on the same kernel paths as for persistent memory - manual configuration
> > > > etc in the kernel.
> > > >
> > > > There is support in ndctl for those enabling flows, so I'd look there
> > > > for more information
> > > >
> > > > Jonathan
> > > >
> > > >  
> > > > >
> > > > >
> > > > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > > > memory.
> > > > >
> > > > > Do I need to pass any particular parameter in the kernel command line?
> > > > >
> > > > > Is there any documentation available? I followed the inputs provided in
> > > > >
> > > > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > > > >
> > > > > Is there any documentation/blog listed?  
> > > >  
> >  

Re: CXL volatile memory is not listed

[Philippe Mathieu-Daudé]

Hi,

Cc'ing Igor and David.

On 9/8/23 00:51, Maverickk 78 wrote:
> Hello,
> 
> I am running qemu-system-x86_64
> 
> qemu-system-x86_64 --version
> QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> 
> qemu-system-x86_64 \
> -m 2G,slots=4,maxmem=4G \
> -smp 4 \
> -machine type=q35,accel=kvm,cxl=on \
> -enable-kvm \
> -nographic \
> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> 
> 
> I was expecting the CXL memory to be listed in "System Ram", the lsmem
> shows only 2G memory which is System RAM, it's not listing the CXL
> memory.

Sounds like a bug. Do you mind reporting at
https://gitlab.com/qemu-project/qemu/-/issues?

Thanks,

Phil.

> Do I need to pass any particular parameter in the kernel command line?
> 
> Is there any documentation available? I followed the inputs provided in
> 
> https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> 
> Is there any documentation/blog listed?
> 

Re: CXL volatile memory is not listed

[David Hildenbrand]

On 10.08.23 12:59, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> Cc'ing Igor and David.
> 
> On 9/8/23 00:51, Maverickk 78 wrote:
>> Hello,
>>
>> I am running qemu-system-x86_64
>>
>> qemu-system-x86_64 --version
>> QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
>>
>> qemu-system-x86_64 \
>> -m 2G,slots=4,maxmem=4G \
>> -smp 4 \
>> -machine type=q35,accel=kvm,cxl=on \
>> -enable-kvm \
>> -nographic \
>> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
>> -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
>> -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
>> -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
>> -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
>>
>>
>> I was expecting the CXL memory to be listed in "System Ram", the lsmem
>> shows only 2G memory which is System RAM, it's not listing the CXL
>> memory.

We are talking about the memory via inside the guest, right?

In the guest, that memory most probably has to be added as "System RAM" 
explicitly using the dax/kmem driver.

https://lwn.net/Articles/922944/

Contains some details on how to use "daxctl reconfigure-device".

-- 
Cheers,

David / dhildenb

Re: CXL volatile memory is not listed

[Maverickk 78]

Thanks Phil, David and Fan

Looks like it was an error from my side due to lack of information
cxl create-region works :)


On Thu, 10 Aug 2023 at 16:29, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> Hi,
>
> Cc'ing Igor and David.
>
> On 9/8/23 00:51, Maverickk 78 wrote:
> > Hello,
> >
> > I am running qemu-system-x86_64
> >
> > qemu-system-x86_64 --version
> > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> >
> > qemu-system-x86_64 \
> > -m 2G,slots=4,maxmem=4G \
> > -smp 4 \
> > -machine type=q35,accel=kvm,cxl=on \
> > -enable-kvm \
> > -nographic \
> > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> >
> >
> > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > shows only 2G memory which is System RAM, it's not listing the CXL
> > memory.
>
> Sounds like a bug. Do you mind reporting at
> https://gitlab.com/qemu-project/qemu/-/issues?
>
> Thanks,
>
> Phil.
>
> > Do I need to pass any particular parameter in the kernel command line?
> >
> > Is there any documentation available? I followed the inputs provided in
> >
> > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> >
> > Is there any documentation/blog listed?
> >
>

Re: CXL volatile memory is not listed

[Fan Ni]

On Wed, Aug 09, 2023 at 04:21:47AM +0530, Maverickk 78 wrote:
> Hello,
>
> I am running qemu-system-x86_64
>
> qemu-system-x86_64 --version
> QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
>
> qemu-system-x86_64 \
> -m 2G,slots=4,maxmem=4G \
> -smp 4 \
> -machine type=q35,accel=kvm,cxl=on \
> -enable-kvm \
> -nographic \
> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
>
>
> I was expecting the CXL memory to be listed in "System Ram", the lsmem
> shows only 2G memory which is System RAM, it's not listing the CXL
> memory.
>
> Do I need to pass any particular parameter in the kernel command line?
>
> Is there any documentation available? I followed the inputs provided in
>
> https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
>
> Is there any documentation/blog listed?

If I remember it correctly, for volatile cxl memory, we need to create a
region and then it will be discovered as system memory and shows up.

Try to create a region with "cxl create-region".

Fan
>

Re: CXL volatile memory is not listed

[Maverickk 78]

Thanks Fan,

cxl create-region works like a charm :)

Since this gets listed as "System Ram(kmem)", I guess the kernel
treats it as regular memory and
allocates it to the applications when needed?
or is there an extra effort needed to make it available for
applications on the host?

On Thu, 10 Aug 2023 at 22:03, Fan Ni <fan.ni@gmx.us> wrote:
>
> On Wed, Aug 09, 2023 at 04:21:47AM +0530, Maverickk 78 wrote:
> > Hello,
> >
> > I am running qemu-system-x86_64
> >
> > qemu-system-x86_64 --version
> > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> >
> > qemu-system-x86_64 \
> > -m 2G,slots=4,maxmem=4G \
> > -smp 4 \
> > -machine type=q35,accel=kvm,cxl=on \
> > -enable-kvm \
> > -nographic \
> > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> >
> >
> > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > shows only 2G memory which is System RAM, it's not listing the CXL
> > memory.
> >
> > Do I need to pass any particular parameter in the kernel command line?
> >
> > Is there any documentation available? I followed the inputs provided in
> >
> > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> >
> > Is there any documentation/blog listed?
>
> If I remember it correctly, for volatile cxl memory, we need to create a
> region and then it will be discovered as system memory and shows up.
>
> Try to create a region with "cxl create-region".
>
> Fan
> >

Re: CXL volatile memory is not listed

[Fan Ni]

On Fri, Aug 11, 2023 at 07:52:25AM +0530, Maverickk 78 wrote:
> Thanks Fan,
>
> cxl create-region works like a charm :)
>
> Since this gets listed as "System Ram(kmem)", I guess the kernel
> treats it as regular memory and
> allocates it to the applications when needed?
> or is there an extra effort needed to make it available for
> applications on the host?
>

Yes. Once it is onlined, you can use it as regular memory.
CXL memory will serve as a zero-CPU memory-only NUMA node.
You can check it with numactl -H.

To use the cxl memory with an app, you can use
numactl --membind=numa_id app_name
#numa_id is the dedicated numa node where cxl memory sits.

One thing to notes, kvm will not work correctly with Qemu emulation when
you try to use cxl memory for an application, so do not enable kvm.

Fan

> On Thu, 10 Aug 2023 at 22:03, Fan Ni <fan.ni@gmx.us> wrote:
> >
> > On Wed, Aug 09, 2023 at 04:21:47AM +0530, Maverickk 78 wrote:
> > > Hello,
> > >
> > > I am running qemu-system-x86_64
> > >
> > > qemu-system-x86_64 --version
> > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > >
> > > qemu-system-x86_64 \
> > > -m 2G,slots=4,maxmem=4G \
> > > -smp 4 \
> > > -machine type=q35,accel=kvm,cxl=on \
> > > -enable-kvm \
> > > -nographic \
> > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> > >
> > >
> > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > memory.
> > >
> > > Do I need to pass any particular parameter in the kernel command line?
> > >
> > > Is there any documentation available? I followed the inputs provided in
> > >
> > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > >
> > > Is there any documentation/blog listed?
> >
> > If I remember it correctly, for volatile cxl memory, we need to create a
> > region and then it will be discovered as system memory and shows up.
> >
> > Try to create a region with "cxl create-region".
> >
> > Fan
> > >

Re: CXL volatile memory is not listed

[Maverickk 78]

Hi Fan

Awesome, thanks for the info!

On Fri, 11 Aug 2023 at 22:19, Fan Ni <fan.ni@gmx.us> wrote:
>
> On Fri, Aug 11, 2023 at 07:52:25AM +0530, Maverickk 78 wrote:
> > Thanks Fan,
> >
> > cxl create-region works like a charm :)
> >
> > Since this gets listed as "System Ram(kmem)", I guess the kernel
> > treats it as regular memory and
> > allocates it to the applications when needed?
> > or is there an extra effort needed to make it available for
> > applications on the host?
> >
>
> Yes. Once it is onlined, you can use it as regular memory.
> CXL memory will serve as a zero-CPU memory-only NUMA node.
> You can check it with numactl -H.
>
> To use the cxl memory with an app, you can use
> numactl --membind=numa_id app_name
> #numa_id is the dedicated numa node where cxl memory sits.
>
> One thing to notes, kvm will not work correctly with Qemu emulation when
> you try to use cxl memory for an application, so do not enable kvm.
>
> Fan
>
> > On Thu, 10 Aug 2023 at 22:03, Fan Ni <fan.ni@gmx.us> wrote:
> > >
> > > On Wed, Aug 09, 2023 at 04:21:47AM +0530, Maverickk 78 wrote:
> > > > Hello,
> > > >
> > > > I am running qemu-system-x86_64
> > > >
> > > > qemu-system-x86_64 --version
> > > > QEMU emulator version 8.0.92 (v8.1.0-rc2-80-g0450cf0897)
> > > >
> > > > qemu-system-x86_64 \
> > > > -m 2G,slots=4,maxmem=4G \
> > > > -smp 4 \
> > > > -machine type=q35,accel=kvm,cxl=on \
> > > > -enable-kvm \
> > > > -nographic \
> > > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> > > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
> > > > -object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
> > > > -device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
> > > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G
> > > >
> > > >
> > > > I was expecting the CXL memory to be listed in "System Ram", the lsmem
> > > > shows only 2G memory which is System RAM, it's not listing the CXL
> > > > memory.
> > > >
> > > > Do I need to pass any particular parameter in the kernel command line?
> > > >
> > > > Is there any documentation available? I followed the inputs provided in
> > > >
> > > > https://lore.kernel.org/linux-mm/Y+CSOeHVLKudN0A6@kroah.com/T/
> > > >
> > > > Is there any documentation/blog listed?
> > >
> > > If I remember it correctly, for volatile cxl memory, we need to create a
> > > region and then it will be discovered as system memory and shows up.
> > >
> > > Try to create a region with "cxl create-region".
> > >
> > > Fan
> > > >