Re: [PATCH 04/11] PCI/IDE: Selective Stream IDE enumeration

[Dan Williams <dan.j.williams@intel.com>]

Jonathan Cameron wrote:
> On Thu, 05 Dec 2024 14:23:39 -0800
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > Link encryption is a new PCIe capability defined by "PCIe 6.2 section
> > 6.33 Integrity & Data Encryption (IDE)". While it is a standalone port
> > and endpoint capability, it is also a building block for device security
> > defined by "PCIe 6.2 section 11 TEE Device Interface Security Protocol
> > (TDISP)". That protocol coordinates device security setup between the
> > platform TSM (TEE Security Manager) and device DSM (Device Security
> > Manager). While the platform TSM can allocate resources like stream-ids
> > and manage keys, it still requires system software to manage the IDE
> > capability register block.
> > 
> > Add register definitions and basic enumeration for a "selective-stream"
> > IDE capability, a follow on change will select the new CONFIG_PCI_IDE
> > symbol. Note that while the IDE specifications defines both a
> > point-to-point "Link" stream and a root-port-to-endpoint "Selective"
> > stream, only "Selective" is considered for now for platform TSM
> > coordination.
> > 
> > Co-developed-by: Alexey Kardashevskiy <aik@amd.com>
> > Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> Some overlap in here with other reviews probably...
> 
> Jonathan
> 
> > ---
> >  drivers/pci/Kconfig           |    3 +
> >  drivers/pci/Makefile          |    1 
> >  drivers/pci/ide.c             |   73 ++++++++++++++++++++++++++++++++++++
> >  drivers/pci/pci.h             |    6 +++
> >  drivers/pci/probe.c           |    1 
> >  include/linux/pci.h           |    5 ++
> >  include/uapi/linux/pci_regs.h |   84 +++++++++++++++++++++++++++++++++++++++++
> >  7 files changed, 172 insertions(+), 1 deletion(-)
> >  create mode 100644 drivers/pci/ide.c
> > 
[..]
> > diff --git a/drivers/pci/ide.c b/drivers/pci/ide.c
> > new file mode 100644
> > index 000000000000..a0c09d9e0b75
> > --- /dev/null
> > +++ b/drivers/pci/ide.c
> > @@ -0,0 +1,73 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/* Copyright(c) 2024 Intel Corporation. All rights reserved. */
> > +
> > +/* PCIe 6.2 section 6.33 Integrity & Data Encryption (IDE) */
> > +
> > +#define dev_fmt(fmt) "PCI/IDE: " fmt
> > +#include <linux/pci.h>
> > +#include "pci.h"
> > +
> > +static int sel_ide_offset(u16 cap, int stream_id, int nr_ide_mem)
> > +{
> > +	return cap + stream_id * PCI_IDE_SELECTIVE_BLOCK_SIZE(nr_ide_mem);
> 
> I'd be tempted to have a define to go from base of the IDE extended cap
> directly to the sel_ide_offset rather than this use of a block based
> offset.  Maybe it ends up too complex though.

Considering other feedback below, I will make this change.


> > +}
> > +
> > +void pci_ide_init(struct pci_dev *pdev)
> > +{
> > +	u16 ide_cap, sel_ide_cap;
> > +	int nr_ide_mem = 0;
> > +	u32 val = 0;
> > +
> > +	if (!pci_is_pcie(pdev))
> > +		return;
> > +
> > +	ide_cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_IDE);
> > +	if (!ide_cap)
> > +		return;
> > +
> > +	/*
> > +	 * Check for selective stream capability from endpoint to root-port, and
> > +	 * require consistent number of address association blocks
> 
> on the EP.
> (for avoidance of confusion).
> 
> Also, from here just seems to mean at the RP and the EP.  Not seting a bus
> walk here to check anything else.  Note I'm not sure we need to but this
> comment is implying a 'from/to' aspect that this code doesn't seem to check.

The from/to aspect is that the ide_cap of endpoints is ignored if the
device's root-port does not have IDE capability.

I will move the comment next to the "if (!rp->ide_cap)" check to make
this clearer.

> > +	 */
> > +	pci_read_config_dword(pdev, ide_cap + PCI_IDE_CAP, &val);
> > +	if ((val & PCI_IDE_CAP_SELECTIVE) == 0)
> > +		return;
> > +
> > +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ENDPOINT) {
> > +		struct pci_dev *rp = pcie_find_root_port(pdev);
> > +
> > +		if (!rp->ide_cap)
> > +			return;
> > +	}
> > +
> > +	if (val & PCI_IDE_CAP_LINK)
> > +		sel_ide_cap = ide_cap + PCI_IDE_LINK_STREAM +
> > +			      (PCI_IDE_CAP_LINK_TC_NUM(val) + 1) *
> > +				      PCI_IDE_LINK_BLOCK_SIZE;
> > +	else
> > +		sel_ide_cap = ide_cap + PCI_IDE_LINK_STREAM;
> Maybe cleaner as
> 	int link_tc_count = 0;
> 	if (val & PCI_IDE_CAP_LINK)
> 		//see suggestion in header to make macro include +1.
> 		link_tc_count = PCI_IDE_CAP_LINK_TC_NUM(val) + 1;
> 
> 	sel_ide_cap = ide_cap + PCI_IDE_LINK_STREAM +
> 		      link_tc_count * PCI_IDE_LINK_BLOCK_SIZE;
> I'm not that bothered either way. Just didn't like that
> ide_cap + PIC_IDE_LINK_STREAM is in both legs.

Makes sense, fixed.

> Or have a macro that always gets you to the selective part without
> using a zero length PCI_IDE_LINK_STREAM block.

Unless it gets multiple use I would open code it in ide.c.

> > +
> > +	for (int i = 0; i < PCI_IDE_CAP_SELECTIVE_STREAMS_NUM(val); i++) {
> > +		if (i == 0) {
> > +			pci_read_config_dword(pdev, sel_ide_cap, &val);
> > +			nr_ide_mem = PCI_IDE_SEL_CAP_ASSOC_NUM(val);
> 
> Yank out and index from 1 for the loop?
> Note though that PCI_IDE_CAP_SELECTIVE_STREAMS_NUM(val) of 1
> means 2 streams so you want <= or just +1 in the macro so the PCI
> header gets to deal with that!

In other review feedback the discussion settled on only shipping offset + masks
in include/uapi/linux/pci_regs.h [1], and put any other logic related to
bitfield.h in ide.c.

[1]: http://lore.kernel.org/67b91d86a48aa_1c530f29431@dwillia2-xfh.jf.intel.com.notmuch


[..]
> > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > index db9b47ce3eef..50811b7655dd 100644
> > --- a/include/linux/pci.h
> > +++ b/include/linux/pci.h
> > @@ -530,6 +530,11 @@ struct pci_dev {
> >  #endif
> >  #ifdef CONFIG_PCI_NPEM
> >  	struct npem	*npem;		/* Native PCIe Enclosure Management */
> > +#endif
> > +#ifdef CONFIG_PCI_IDE
> > +	u16		ide_cap;	/* Link Integrity & Data Encryption */
> > +	u16		sel_ide_cap;	/* - Selective Stream register block */
> 
> I'd not call it cap as people will go looking for a selective IDE extended capability.
> I'm a little dubious about it being necessary vs a helper function that grabs
> the necessary count info directly from the device.

I was trying to avoid extra config cycles in the common case, but there
is no precedent for caching extra offsets in 'struct pci_dev'.

I am ok to drop sel_ide_cap.

> 
> > +	int		nr_ide_mem;	/* - Address range limits for streams */
> >  #endif
> >  	u16		acs_cap;	/* ACS Capability offset */
> >  	u8		supported_speeds; /* Supported Link Speeds Vector */
> > diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> > index 1601c7ed5fab..9635b27d2485 100644
> > --- a/include/uapi/linux/pci_regs.h
> > +++ b/include/uapi/linux/pci_regs.h
> > @@ -748,7 +748,8 @@
> >  #define PCI_EXT_CAP_ID_NPEM	0x29	/* Native PCIe Enclosure Management */
> >  #define PCI_EXT_CAP_ID_PL_32GT  0x2A    /* Physical Layer 32.0 GT/s */
> >  #define PCI_EXT_CAP_ID_DOE	0x2E	/* Data Object Exchange */
> > -#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_DOE
> > +#define PCI_EXT_CAP_ID_IDE	0x30    /* Integrity and Data Encryption */
> > +#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_IDE
> >  
> >  #define PCI_EXT_CAP_DSN_SIZEOF	12
> >  #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
> > @@ -1213,4 +1214,85 @@
> >  #define PCI_DVSEC_CXL_PORT_CTL				0x0c
> >  #define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR		0x00000001
> >  
> > +/* Integrity and Data Encryption Extended Capability */
> > +#define PCI_IDE_CAP			0x4
> > +#define  PCI_IDE_CAP_LINK		0x1  /* Link IDE Stream Supported */
> > +#define  PCI_IDE_CAP_SELECTIVE		0x2  /* Selective IDE Streams Supported */
> > +#define  PCI_IDE_CAP_FLOWTHROUGH	0x4  /* Flow-Through IDE Stream Supported */
> > +#define  PCI_IDE_CAP_PARTIAL_HEADER_ENC 0x8  /* Partial Header Encryption Supported */
> > +#define  PCI_IDE_CAP_AGGREGATION	0x10 /* Aggregation Supported */
> > +#define  PCI_IDE_CAP_PCRC		0x20 /* PCRC Supported */
> > +#define  PCI_IDE_CAP_IDE_KM		0x40 /* IDE_KM Protocol Supported */
> 
> Looks like 3.2 has a bit 7 defined as well.  Selective IDE for configuration requests supported.
> Probably worth adding that.

Might as well.

> 
> > +#define  PCI_IDE_CAP_ALG(x)		(((x) >> 8) & 0x1f) /* Supported Algorithms */
> > +#define  PCI_IDE_CAP_ALG_AES_GCM_256	0    /* AES-GCM 256 key size, 96b MAC */
> > +#define  PCI_IDE_CAP_LINK_TC_NUM(x)	(((x) >> 13) & 0x7) /* Link IDE TCs */
> Maybe add 1 here as the macro name kind of implies it is returning the number of link IDE TCs
> rather than 1 less that that. It is a little tricky given the spec calls this field "Number of"
> 
> > +#define  PCI_IDE_CAP_SELECTIVE_STREAMS_NUM(x)	(((x) >> 16) & 0xff) /* Selective IDE Streams */
> 
> Similar here. I'm not sure what precedence we have int his file. I can't immediately see any
> either way. 
> 
> > +#define  PCI_IDE_CAP_SELECTIVE_STREAMS_MASK	0xff0000
> Why have the mask if you are providing the macro above to get the value?

A mix of copying from the SEV-TIO vs TDX Connect RFCs. Per other
feedback, I have now resolved to only defines masks and offsets and drop
the decorated helpers that are open coding bitmask.h.

It turns out that __GENMASK is available in uapi/linux/bits.h, so I will
switch to that.

> > +#define  PCI_IDE_CAP_TEE_LIMITED	0x1000000 /* TEE-Limited Stream Supported */
> > +#define PCI_IDE_CTL			0x8
> > +#define  PCI_IDE_CTL_FLOWTHROUGH_IDE	0x4	/* Flow-Through IDE Stream Enabled */
> > +#define PCI_IDE_LINK_STREAM		0xc
> I couldn't find specific precedence for this but my gut would say add a _0 postfix
> to indicate it's the first of a number of these.
> All the similar cases seem to explicitly enumerate _0, _1 etc which makes little
> sense here.
> 
> > +#define PCI_IDE_LINK_BLOCK_SIZE		8
> > +/* Link IDE Stream block, up to PCI_IDE_CAP_LINK_TC_NUM */
> > +/* Link IDE Stream Control Register */
> I'd expect a _0 define for the first ctrl and one for the first status.
> 
> Then index each register via
> PCI_IDE_LINK_CTL_0 + i * PCIE_IDE_LINK_BLOCK_SIZE
> PCI_IDE_LINK_STS_0 + i * PCIE_IDE_LINK_BLOCK_SIZE
> 
> Again, not immediately seeing precedence, but having register field defines without
> a register address define (even a constructed one as will be relevant
> for the selective IDE stream blocks) seems odd to me.

I will add the _0, but skip the rest for now. There is no precedence I
can see for the amount of degrees of freedom in this IDE register block for
the location of the selective registers, and Linux does not currently
have a use case for Link IDE. I imagine any Link Register Block walking
will live in ide.c. I.e. given Selective Stream block offset calculation
lives in ide.c might as well do the same for Link IDE when/if needed.

> 
> > +#define  PCI_IDE_LINK_CTL_EN		 0x1	/* Link IDE Stream Enable */
> > +#define  PCI_IDE_LINK_CTL_TX_AGGR_NPR(x) (((x) >> 2) & 0x3) /* Tx Aggregation Mode NPR */
> > +#define  PCI_IDE_LINK_CTL_TX_AGGR_PR(x)	 (((x) >> 4) & 0x3) /* Tx Aggregation Mode PR */
> > +#define  PCI_IDE_LINK_CTL_TX_AGGR_CPL(x) (((x) >> 6) & 0x3) /* Tx Aggregation Mode CPL */
> > +#define  PCI_IDE_LINK_CTL_PCRC_EN	 0x100	/* PCRC Enable */
> > +#define  PCI_IDE_LINK_CTL_PART_ENC(x)	 (((x) >> 10) & 0xf)  /* Partial Header Encryption Mode */
> > +#define  PCI_IDE_LINK_CTL_ALG(x)	 (((x) >> 14) & 0x1f) /* Selected Algorithm */
> Perhaps nice to throw in a reference to the supported algs list above.

Ok

> 
> > +#define  PCI_IDE_LINK_CTL_TC(x)		 (((x) >> 19) & 0x7)  /* Traffic Class */
> > +#define  PCI_IDE_LINK_CTL_ID(x)		 (((x) >> 24) & 0xff) /* Stream ID */
> > +#define  PCI_IDE_LINK_CTL_ID_MASK	 0xff000000
> > +
> > +
> > +/* Link IDE Stream Status Register */
> > +#define  PCI_IDE_LINK_STS_STATUS(x)	((x) & 0xf) /* Link IDE Stream State */
> > +#define  PCI_IDE_LINK_STS_RECVD_INTEGRITY_CHECK	0x80000000 /* Received Integrity Check Fail Msg */
> 
> 
> I'd put some white space here.

Ok

> 
> > +/* Selective IDE Stream block, up to PCI_IDE_CAP_SELECTIVE_STREAMS_NUM */
> > +#define PCI_IDE_SELECTIVE_BLOCK_SIZE(x)  (20 + 12 * (x))
> 
> Probably want a better name than 'x' for that parameter as it's not
> immediately obvious what it is. (number of IDE address association
> register blocks).
> Also that 12 probably wants a define. It's used a few times.

Ok

> 
> > +/* Selective IDE Stream Capability Register */
> > +#define  PCI_IDE_SEL_CAP		 0
> > +#define  PCI_IDE_SEL_CAP_ASSOC_NUM(x)	 ((x) & 0xf) /* Address Association Register Blocks Number */
> > +#define  PCI_IDE_SEL_CAP_ASSOC_MASK	 0xf
> 
> If the mask make sense to keep at all would be good to build
> the macro above using it.

Dropped the macro, kept the mask.

> 
> > +/* Selective IDE Stream Control Register */
> > +#define  PCI_IDE_SEL_CTL		 4
> > +#define   PCI_IDE_SEL_CTL_EN		 0x1	/* Selective IDE Stream Enable */
> > +#define   PCI_IDE_SEL_CTL_TX_AGGR_NPR(x) (((x) >> 2) & 0x3) /* Tx Aggregation Mode NPR */
> > +#define   PCI_IDE_SEL_CTL_TX_AGGR_PR(x)	 (((x) >> 4) & 0x3) /* Tx Aggregation Mode PR */
> > +#define   PCI_IDE_SEL_CTL_TX_AGGR_CPL(x) (((x) >> 6) & 0x3) /* Tx Aggregation Mode CPL */
> > +#define   PCI_IDE_SEL_CTL_PCRC_EN	 0x100	/* PCRC Enable */
> > +#define   PCI_IDE_SEL_CTL_CFG_EN	 0x200	/* Selective IDE for Configuration Requests */
> > +#define   PCI_IDE_SEL_CTL_PART_ENC(x)	 (((x) >> 10) & 0xf)  /* Partial Header Encryption Mode */
> This is a control register. Seems likely we'll mostly be writing these.
> So how useful is it to provide just a read macro?
> Maybe I'm missing something!

I agree and this matches other feedback prompting the "masks only"
stance.


> > +#define   PCI_IDE_SEL_CTL_ALG(x)	 (((x) >> 14) & 0x1f) /* Selected Algorithm */
> > +#define   PCI_IDE_SEL_CTL_TC(x)		 (((x) >> 19) & 0x7)  /* Traffic Class */
> > +#define   PCI_IDE_SEL_CTL_DEFAULT	 0x400000 /* Default Stream */
> > +#define   PCI_IDE_SEL_CTL_TEE_LIMITED	 (1 << 23) /* TEE-Limited Stream */
> 
> Why this one as a shift and all the rest as explicit hex values?

Fixed.

> 
> > +#define   PCI_IDE_SEL_CTL_ID_MASK	 0xff000000
> > +#define   PCI_IDE_SEL_CTL_ID_MAX	 255
> > +/* Selective IDE Stream Status Register */
> > +#define  PCI_IDE_SEL_STS		 8
> > +#define   PCI_IDE_SEL_STS_STATUS(x)	((x) & 0xf) /* Selective IDE Stream State */
> > +#define   PCI_IDE_SEL_STS_RECVD_INTEGRITY_CHECK	0x80000000 /* Received Integrity Check Fail Msg */
> > +/* IDE RID Association Register 1 */
> > +#define  PCI_IDE_SEL_RID_1		 12
> > +#define   PCI_IDE_SEL_RID_1_LIMIT_MASK	 0xffff00
> > +/* IDE RID Association Register 2 */
> > +#define  PCI_IDE_SEL_RID_2		 16
> > +#define   PCI_IDE_SEL_RID_2_VALID	 0x1
> > +#define   PCI_IDE_SEL_RID_2_BASE_MASK	 0x00ffff00
> 
> Why leading zeros on this one?

Fixed.

> 
> > +#define   PCI_IDE_SEL_RID_2_SEG_MASK	 0xff000000
> > +/* Selective IDE Address Association Register Block, up to PCI_IDE_SEL_CAP_ASSOC_NUM */
> > +#define  PCI_IDE_SEL_ADDR_1(x)		     (20 + (x) * 12)
> > +#define   PCI_IDE_SEL_ADDR_1_VALID	     0x1
> > +#define   PCI_IDE_SEL_ADDR_1_BASE_LOW_MASK   0x000fff0
> 
> more leading zeros which doesn't seem consistent. Also, as Alexey
> pointed out value is wrong as that's 4 bits in not 8.
> 
> 
> > +#define   PCI_IDE_SEL_ADDR_1_BASE_LOW_SHIFT  20
> 8?

This was defining how much to shift the lower 32-bits of an address to
feed this value. Moved all that detail to ide.c

> 
> > +#define   PCI_IDE_SEL_ADDR_1_LIMIT_LOW_MASK  0xfff0000
> > +#define   PCI_IDE_SEL_ADDR_1_LIMIT_LOW_SHIFT 20
> Also missing a zero (Alexey got this one as well I see)

Got it, thanks for going through all that!