RFC: x86 memory map, where to put CXL ranges?

RFC: x86 memory map, where to put CXL ranges?

[Jonathan Cameron via]

Hi All,

For CXL emulation we require a couple of types of memory range that
are then provided to the OS via the CEDT ACPI table.

1) CXL Host Bridge Structures point to CXL Host Bridge Component Registers.
Small regions for each CXL Host bridge that are mapped into the memory space.
64k each.  In theory we may have a huge number of these but in reality I
think 16 will do for any reasonable system.

2) CXL Fixed Memory Window Structures (CFMWS)
Large PA space ranges (multiple TB) to which various CXL devices can be assigned
and their address decoders appropriately programmed.
Each such CFMWS will have particular characteristics such as interleaving across
multiple host bridges.  The can potentially be huge but are a system
characteristic.  For emulation purposes it won't matter if they move around
dependent on what else is the machine has configured. So I'd like to
just configure their size rather than fully specify them at the command line
and possibly clash on PA space with something else.  Alternatively could
leave them as fully specified at the command line (address and size) and just
error out if the hit memory already in use for something else.

Now unfortunately there are no systems out there yet that we can just
copy the memory map from...

Coming form an Arm background I have only a vague idea of how this should be
done for x86 so apologies if it is a stupid question.

My current approach is to put these above device_memory and moving
the pci hole up appropriately.

Is that the right choice?

On Arm I currently have the Host Bridge Structures low down in the MemMap and the CFMWS
can go above the device memory.  Comments on that also welcome.

In Ben's RFC the host bridge component register location was marked as a TODO
and a arbitrary address used in the meantime so time to figure out how to clean
that up.

Thanks,

Jonathan

Re: RFC: x86 memory map, where to put CXL ranges?

[Alex Bennée]

Jonathan Cameron <Jonathan.Cameron@Huawei.com> writes:

> Hi All,
>
> For CXL emulation we require a couple of types of memory range that
> are then provided to the OS via the CEDT ACPI table.
>
> 1) CXL Host Bridge Structures point to CXL Host Bridge Component Registers.
> Small regions for each CXL Host bridge that are mapped into the memory space.
> 64k each.  In theory we may have a huge number of these but in reality I
> think 16 will do for any reasonable system.
>
> 2) CXL Fixed Memory Window Structures (CFMWS)
> Large PA space ranges (multiple TB) to which various CXL devices can be assigned
> and their address decoders appropriately programmed.
> Each such CFMWS will have particular characteristics such as interleaving across
> multiple host bridges.  The can potentially be huge but are a system
> characteristic.  For emulation purposes it won't matter if they move around
> dependent on what else is the machine has configured. So I'd like to
> just configure their size rather than fully specify them at the command line
> and possibly clash on PA space with something else.  Alternatively could
> leave them as fully specified at the command line (address and size) and just
> error out if the hit memory already in use for something else.
>
> Now unfortunately there are no systems out there yet that we can just
> copy the memory map from...
>
> Coming form an Arm background I have only a vague idea of how this should be
> done for x86 so apologies if it is a stupid question.
>
> My current approach is to put these above device_memory and moving
> the pci hole up appropriately.

Which board model would be be talking about here? virt? Or maybe we need
a new one?

If it's virt I would look at extended_memmap which floats above the
configured RAM size and means less shuffling around of the relatively
crowded lower address space.

I have no idea about how this is handled on x86 though.

> Is that the right choice?
>
> On Arm I currently have the Host Bridge Structures low down in the MemMap and the CFMWS
> can go above the device memory.  Comments on that also welcome.
>
> In Ben's RFC the host bridge component register location was marked as a TODO
> and a arbitrary address used in the meantime so time to figure out how to clean
> that up.
>
> Thanks,
>
> Jonathan


-- 
Alex Bennée

Re: RFC: x86 memory map, where to put CXL ranges?

[Jonathan Cameron via]

On Thu, 09 Dec 2021 14:19:59 +0000
Alex Bennée <alex.bennee@linaro.org> wrote:

> Jonathan Cameron <Jonathan.Cameron@Huawei.com> writes:
> 
> > Hi All,
> >
> > For CXL emulation we require a couple of types of memory range that
> > are then provided to the OS via the CEDT ACPI table.
> >
> > 1) CXL Host Bridge Structures point to CXL Host Bridge Component Registers.
> > Small regions for each CXL Host bridge that are mapped into the memory space.
> > 64k each.  In theory we may have a huge number of these but in reality I
> > think 16 will do for any reasonable system.
> >
> > 2) CXL Fixed Memory Window Structures (CFMWS)
> > Large PA space ranges (multiple TB) to which various CXL devices can be assigned
> > and their address decoders appropriately programmed.
> > Each such CFMWS will have particular characteristics such as interleaving across
> > multiple host bridges.  The can potentially be huge but are a system
> > characteristic.  For emulation purposes it won't matter if they move around
> > dependent on what else is the machine has configured. So I'd like to
> > just configure their size rather than fully specify them at the command line
> > and possibly clash on PA space with something else.  Alternatively could
> > leave them as fully specified at the command line (address and size) and just
> > error out if the hit memory already in use for something else.
> >
> > Now unfortunately there are no systems out there yet that we can just
> > copy the memory map from...
> >
> > Coming form an Arm background I have only a vague idea of how this should be
> > done for x86 so apologies if it is a stupid question.
> >
> > My current approach is to put these above device_memory and moving
> > the pci hole up appropriately.  
> 
> Which board model would be be talking about here? virt? Or maybe we need
> a new one?

Initially at least the plan is virt because all the acpi table building support
is there which makes things nice and easy.  It's not very invasive and there
has to be a machine level cxl=on to enable it so shouldn't be a maintenance problem
and doesn't justify new board model.

> 
> If it's virt I would look at extended_memmap which floats above the
> configured RAM size and means less shuffling around of the relatively
> crowded lower address space.

I'll give that a go.

There are two types of regions to find space for.
1) Root complex control registers.  Those can go in the memory map.  Currently
I have them fitted in a gap in the low memory map but I'll move them to the extended
one as doesn't matter if they are at a fixed address or not.
2) Fixed memory windows described in CEDT/CFMWS entries.
These are less obvious.  They are PA ranges with associated interleaving
(across host bridges) and peformance properties (related to platform QoS
control etc). They act as PA ranges into which we can assign particular devices.

For an initial patch set I plan to support no interleaving and just have
one such region per host bridge (currently pxb instance).
For these regions, my current choice is to put them above device_memory.
So at the command line you can specify a set of region sizes and we
then lay them out appropriately and describe those in CFMWS entries.
Ben's earlier code had these fully specified as base + size, but I don't
think there is any advantage in doing that as the info is presented to
the OS etc anyway so it doesn't matter if it changes. If it is
figured out at runtime we will have less problem with accidental clashes.

> 
> I have no idea about how this is handled on x86 though.

I'll go with the show the code for that :)

Thanks,

Jonathan

> 
> > Is that the right choice?
> >
> > On Arm I currently have the Host Bridge Structures low down in the MemMap and the CFMWS
> > can go above the device memory.  Comments on that also welcome.
> >
> > In Ben's RFC the host bridge component register location was marked as a TODO
> > and a arbitrary address used in the meantime so time to figure out how to clean
> > that up.
> >
> > Thanks,
> >
> > Jonathan  
> 
>