All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v12 0/8] DCD Prep Series
@ 2026-07-31  8:48 Anisa Su
  2026-07-31  8:48 ` [PATCH v12 1/8] cxl/mbox: Flag support for Dynamic Capacity Devices (DCD) Anisa Su
                   ` (7 more replies)
  0 siblings, 8 replies; 13+ messages in thread
From: Anisa Su @ 2026-07-31  8:48 UTC (permalink / raw)
  To: linux-cxl
  Cc: nvdimm, linux-kernel, Dave Jiang, Alison Schofield,
	Jonathan Cameron, Fan Ni, Li Ming, Vishal Verma, Davidlohr Bueso,
	Ira Weiny, Benjamin Cheatham, Wonjae Lee, Junhee Park, Heesoo Kim,
	Anisa Su

These are the first eight patches of the Dynamic Capacity Device (DCD)
series, split from v11 and posted on their own: DCD enumeration, CDAT handling
for DC partitions, the event interrupt plumbing, and two prepatory refactors.
They stand alone, and keep DCD disabled. They are not dependent on the
rest of the series, so I think they are safe enough to pick up ahead
of the rest.

Patch 1 detects the 48h command set and then leaves mds->dcd_supported
false. The flag is only turned on by the event handling patch later
in the full series, so every DCD specific path added here is shut off at
runtime: cxl_configure_dcd() never runs, no DC partition is created,
no DCD interrupt is armed, and the CXL_PARTMODE_DYNAMIC_RAM_1 paths are
unreachable.  The intent is to land the enumeration and plumbing while
the extent to DAX pipeline is still under review.

Based on v7.2-rc5.

GitHub Branch:
https://github.com/anisa-su993/anisa-linux-kernel/tree/dcd-prep-series

v11 -> v12 Changes:
===================
 - Patch 1: cxl/mbox: Flag support for Dynamic Capacity Devices (DCD)
        - now leaves DCD disabled (mds->dcd_supported = false)

 - Patch 2: cxl/mem: Read dynamic capacity configuration from the device
        - mbox.c: bound the Get DC Config request by the negotiated
          mailbox payload. Requesting all 8 partitions needs 328 bytes,
          which exceeds a spec-minimum 256-byte mailbox and fails -E2BIG,
          disabling DCD. Cap partition_count per call and read the rest
          via the existing loop (Sashiko)

        - cxlmem.h/mbox.c: bump DC config comment references from CXL 3.2
          to CXL r4.0 (Tables 8-178/179/180), matching the commit log (Dave)

 - Patch 3: cxl/cdat: Gather DSMAS data for DCD partitions\
        - No functional change. Just added the tags below
        + Co-developed-by: Anisa Su (for shareable support)
        + Reviewed-by: Dave Jiang

 - Patch 6:
        - pci.c: size the Set Event Interrupt Policy payload by the
          device's policy layout (Get reply length) instead of assuming
          based on DCD command support; a CXL 3.0+ non-DCD device can have
          the dcd_settings policy bit but not support DCD commands (Benjamin Cheatham).
        - link to report: https://lore.kernel.org/linux-cxl/20260625112638.550691-1-anisa.su@samsung.com/T/#m09b4512dbde6e1f47a27f28c44e23d83a2d628e4

        - pci.c: add cxl_event_drain_mask() helper to set drain mask
          (standard logs when native_cxl, DCD when supported). Shared by
          cxl_event_thread() and the initial drain in cxl_event_config() to
          address the 2 issues reported by Sashiko below:
          1. If cxl_event_req_irq() fails for the DCD log, cxl_irqsetup()
            calls cxl_disable_dcd() — which clears the driver's flag but
            leaves the device armed for DCD interrupts. Then cxl_event_thread(),
            whose drain mask still included the DCD bit, would see
            CXLDEV_EVENT_STATUS_DCD set while cxl_mem_get_event_records()
            refused to drain that log (it checks cxl_dcd_supported(), now false),
            leaving the bit permanently set and spinning the
            do { } while (status) loop forever.
          2. The probe-time drain was gated on native_cxl, but the driver owns the
            DCD log regardless of who owns standard error handling (firmware cannot take DCD).
            So on a BIOS-first platform with a DCD device, the pre-existing DCD events were
            never drained at probe. Because the interrupts are edge-triggered, that already-set
            status bit could strand future notifications
          > link to report: https://lore.kernel.org/linux-cxl/20260625112638.550691-1-anisa.su@samsung.com/T/#m9141704ffeadd08ea86d16121575c4b5c400d107

 - misc. checkpatch and comment cleanups across the set

 - Tested-by: from Wonjae Lee, Junhee Park and Heesoo Kim on all 8 patches

The tags apply to the v11 series but I carried the tags over because
the changes detailed above (256B mailbox size, card with dcd_settings policy
bit but doesn't support 48xx cmds) are not relevant to the test configuration,
which has a sufficiently large mailbox size, and has both the
dcd_settings policy bit + supports the 48xx commands.
But if needed, this can be re-tested on HW. I thought
it might not be very meaningful since DCD is turned off.

Instead, I emulated a 256B mailbox with QEMU and a 3.0 non-DCD device to
check that the fixes work. Detailed below.

v11 (full series):
https://lore.kernel.org/linux-cxl/20260625112638.550691-1-anisa.su@samsung.com/

What changes for devices without DCD
====================================
Code that touches non-DCD devices:

 - ram_size and ram qos_class locate the RAM partition by mode instead of
   assuming index 0; cxl_pmem_size() becomes cxl_part_size().

 - CXL_NR_PARTITIONS_MAX goes from 2 to 3, and each partition records its
   DSMAS handle and shareable flag from CDAT. Both fields are written
   here but not consumed until the rest of the patchset.

Review status
=============
Patches 1, 2 and 6 carry no Reviewed-by: the DCD capability flag, the
Get DC Config read, and the event interrupt configuration, so review
there is especially welcome.  Patches 3 and 8 picked up Dave Jiang's
Reviewed-by since v11; the rest carry their v11 tags unchanged.

Patch organization
==================
 1-3: enumerate DCD command support, read the DC configuration, and pick
      up DSMAS data for DC partitions.
 4-6: split event msgnum configuration from irq setup, factor out the
      interrupt policy check, then configure the DCD event interrupt.
 7-8: prepatory refactors -- return the endpoint decoder from the region
      search, and enforce partition order while collapsing the ram/pmem
      specific partition helpers into one.

Testing
=======
As I mentioned above, the full v11 series was tested on shareable DCD hw
on GNR platform, but since the test configuration doesn't apply to
the changes made in this version, I quickly tweaked QEMU to emulate a
256B mailbox (set CXL_MAILBOX_PAYLOAD_SHIFT = 8) and set the minimum
policy length from 4 to 5 (to include dcd_settings).

The relevant conclusion from the HW testing on v11 is that adding
and releasing extents succeeded, so the event interrupts were working.
Additionally, the device is shareable and an untagged (null uuid)
extent was rejected, so partition data is probably read correctly.

Also used ndctl-test-runner to run cxl_test to test for regressions.
The NDCTL branch tested against is pmem/pending:
https://github.com/anisa-su993/ndctl-test-runner/actions/runs/30613886000

QEMU test results: interrupt policy sizing and Get DC Config bounding
=====================================================================
QEMU:   cxl-type3, two local modifications described per test

Test 1: Set Event Interrupt Policy against a strict device
----------------------------------------------------------
Models the card Benjamin Cheatham reported: CXL 3.0+, exposes the DCD
interrupt-policy field, does NOT advertise the 48h DCD command set.

QEMU modification -- make the device reject the 4 byte policy payload:

    --- a/include/hw/cxl/cxl_events.h
    +++ b/include/hw/cxl/cxl_events.h
    -#define CXL_EVENT_INT_SETTING_MIN_LEN 4
    +#define CXL_EVENT_INT_SETTING_MIN_LEN 5

Topology: plain cxl-type3 (no dc-regions), so no 48h commands appear in
the CEL and the driver leaves DCD unsupported.

With the fix (payload sized from the device's Get reply):

    Event config : OS DCD not supported
    memdev count:  1

The device probes, and the Set succeeds because the driver echoes back
the 5-byte layout the device declared in its Get reply, even though DCD
command support is off.

Without the fix -- same topology
(.size_in = CXL_EVENT_INT_POLICY_BASE_SIZE, i.e. always 4 bytes):

    [   13.635070] cxl_pci:__cxl_pci_mbox_send_cmd:346: cxl_pci 0000:0d:00.0: Mailbox operation had an error: invalid payload length
    [   13.638125] cxl_pci 0000:0d:00.0: Failed to set event interrupt policy : -6
    memdev count:  0

The failure is not cosmetic: cxl_event_config() propagates the error and
cxl_pci_probe() returns it, so the memory device does not attach at all.
This reproduces the reported failure and confirms the fix addresses it.


Test 2: Get DC Config on a spec-minimum mailbox
-----------------------------------------------
CXL r3.1 allows a 256-byte mailbox payload.  Before the v12 change the
driver asked for all CXL_MAX_DC_PARTITIONS (8) partitions in one Get DC
Config -- 8 + 8 * 40 = 328 bytes -- which cxl_internal_send_cmd() rejects
with -E2BIG before the command reaches the device, leaving DCD silently
unavailable.

QEMU modification -- shrink the mailbox to the spec minimum:

    --- a/include/hw/cxl/cxl_device.h
    +++ b/include/hw/cxl/cxl_device.h
    -#define CXL_MAILBOX_PAYLOAD_SHIFT 11        /* 2048 bytes */
    +#define CXL_MAILBOX_PAYLOAD_SHIFT 8         /*  256 bytes */

2a. Single DC region -- PASSES

    [  370.443819] cxl_pci_setup_mailbox: Mailbox payload sized 256
    [  370.574517] cxl_dc_check: DC partition 0 start 0 size 12884901888 blk_size: 2097152
    [  370.579817] cxl_dev_dc_identify: Returning partition 0 0 size 12884901888
    [  370.584529] cxl_configure_dcd: Adding dynamic ram partition 1; 0 size 12884901888
    [  370.588750] add_dpa_res: DPA(dynamic_ram_1): [mem 0x00000000-0x2ffffffff flags 0x200]

    # cat /sys/bus/cxl/devices/mem0/dynamic_ram_1/size
    0x300000000

cxl_dc_check() and cxl_dev_dc_identify() only reach those dev_dbg()s
after cxl_get_dc_config() returns successfully, so the read completed
rather than failing -E2BIG.  Exactly one "Sending command: 0x4800"
appears: the device has one region, so the per-call cap is exercised but
the start_partition continuation loop is not.

2b. Eight DC regions -- PASSES

Re-run with num-dc-regions=8 to force more partitions than fit in one
payload to test that the loop for retrieving all paritions is working.

    Mailbox payload sized 256
    "Sending command: 0x4800" count: 2

    [   13.037634] cxl_pci:__cxl_pci_mbox_send_cmd:263: cxl_pci 0000:0d:00.0: Sending command: 0x4800
    [   13.039762] cxl_pci:cxl_pci_mbox_wait_for_doorbell:74: cxl_pci 0000:0d:00.0: Doorbell wait took 0ms
    [   13.042483] cxl_core:cxl_get_dc_config:2166: cxl_pci 0000:0d:00.0: Read 5/8 DC partitions
    [   13.044020] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 0 start 0 size 1610612736 blk_size: 2097152
    [   13.046735] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 1 start 1610612736 size 1610612736 blk_size: 2097152
    [   13.049732] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 2 start 3221225472 size 1610612736 blk_size: 2097152
    [   13.053862] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 3 start 4831838208 size 1610612736 blk_size: 2097152
    [   13.056833] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 4 start 6442450944 size 1610612736 blk_size: 2097152
    [   13.059801] cxl_pci:__cxl_pci_mbox_send_cmd:263: cxl_pci 0000:0d:00.0: Sending command: 0x4800
    [   13.062293] cxl_pci:cxl_pci_mbox_wait_for_doorbell:74: cxl_pci 0000:0d:00.0: Doorbell wait took 0ms
    [   13.064680] cxl_core:cxl_get_dc_config:2166: cxl_pci 0000:0d:00.0: Read 3/8 DC partitions
    [   13.066897] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 5 start 8053063680 size 1610612736 blk_size: 2097152
    [   13.072694] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 6 start 9663676416 size 1610612736 blk_size: 2097152
    [   13.076634] cxl_core:cxl_dc_check:2111: cxl_pci 0000:0d:00.0: DC partition 7 start 11274289152 size 1610612736 blk_size: 2097152
    [   13.080084] cxl_core:cxl_dev_dc_identify:2256: cxl_pci 0000:0d:00.0: Returning partition 0 0 size 1610612736

Two Get DC Config commands rather than one: the device holds eight
partitions and only five fit per call, so start_partition advances and
the remaining three are read in a second call. 

Ira Weiny (8):
  cxl/mbox: Flag support for Dynamic Capacity Devices (DCD)
  cxl/mem: Read dynamic capacity configuration from the device
  cxl/cdat: Gather DSMAS data for DCD partitions
  cxl/events: Split event msgnum configuration from irq setup
  cxl/pci: Factor out interrupt policy check
  cxl/mem: Configure dynamic capacity interrupts
  cxl/core: Return endpoint decoder information from region search
  cxl/core: Enforce partition order/simplify partition calls

 drivers/cxl/core/cdat.c   |  13 ++
 drivers/cxl/core/core.h   |   6 +-
 drivers/cxl/core/hdm.c    |  14 +-
 drivers/cxl/core/mbox.c   | 286 +++++++++++++++++++++++++++++++++++++-
 drivers/cxl/core/memdev.c |  36 ++---
 drivers/cxl/core/region.c |   8 +-
 drivers/cxl/cxl.h         |   4 +-
 drivers/cxl/cxlmem.h      |  88 +++++++++++-
 drivers/cxl/mem.c         |   2 +-
 drivers/cxl/pci.c         | 161 ++++++++++++++++-----
 include/cxl/cxl.h         |   7 +-
 11 files changed, 546 insertions(+), 79 deletions(-)


base-commit: f5098b6bae761e346ebcd9da7f95622c04733cff
-- 
2.43.0


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2026-07-31  9:04 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-31  8:48 [PATCH v12 0/8] DCD Prep Series Anisa Su
2026-07-31  8:48 ` [PATCH v12 1/8] cxl/mbox: Flag support for Dynamic Capacity Devices (DCD) Anisa Su
2026-07-31  8:48 ` [PATCH v12 2/8] cxl/mem: Read dynamic capacity configuration from the device Anisa Su
2026-07-31  9:01   ` sashiko-bot
2026-07-31  8:48 ` [PATCH v12 3/8] cxl/cdat: Gather DSMAS data for DCD partitions Anisa Su
2026-07-31  9:02   ` sashiko-bot
2026-07-31  8:48 ` [PATCH v12 4/8] cxl/events: Split event msgnum configuration from irq setup Anisa Su
2026-07-31  8:48 ` [PATCH v12 5/8] cxl/pci: Factor out interrupt policy check Anisa Su
2026-07-31  8:48 ` [PATCH v12 6/8] cxl/mem: Configure dynamic capacity interrupts Anisa Su
2026-07-31  9:04   ` sashiko-bot
2026-07-31  8:48 ` [PATCH v12 7/8] cxl/core: Return endpoint decoder information from region search Anisa Su
2026-07-31  9:01   ` sashiko-bot
2026-07-31  8:48 ` [PATCH v12 8/8] cxl/core: Enforce partition order/simplify partition calls Anisa Su

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.