* [PATCH v7 18/24] iommu/arm-smmu-v3: Add support for Hardware Translation Table Update
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
If the SMMU supports it and the kernel was built with HTTU support,
enable hardware update of access and dirty flags. This is essential for
shared page tables, to reduce the number of access faults on the fault
queue. Normal DMA with io-pgtables doesn't currently use the access or
dirty flags.
We can enable HTTU even if CPUs don't support it, because the kernel
always checks for HW dirty bit and updates the PTE flags atomically.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/arm-smmu-v3.c | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 1386d4d2bc60..6a368218f54c 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -58,6 +58,8 @@
#define IDR0_ASID16 (1 << 12)
#define IDR0_ATS (1 << 10)
#define IDR0_HYP (1 << 9)
+#define IDR0_HD (1 << 7)
+#define IDR0_HA (1 << 6)
#define IDR0_BTM (1 << 5)
#define IDR0_COHACC (1 << 4)
#define IDR0_TTF GENMASK(3, 2)
@@ -311,6 +313,9 @@
#define CTXDESC_CD_0_TCR_IPS GENMASK_ULL(34, 32)
#define CTXDESC_CD_0_TCR_TBI0 (1ULL << 38)
+#define CTXDESC_CD_0_TCR_HA (1UL << 43)
+#define CTXDESC_CD_0_TCR_HD (1UL << 42)
+
#define CTXDESC_CD_0_AA64 (1UL << 41)
#define CTXDESC_CD_0_S (1UL << 44)
#define CTXDESC_CD_0_R (1UL << 45)
@@ -663,6 +668,8 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_E2H (1 << 16)
#define ARM_SMMU_FEAT_BTM (1 << 17)
#define ARM_SMMU_FEAT_SVA (1 << 18)
+#define ARM_SMMU_FEAT_HA (1 << 19)
+#define ARM_SMMU_FEAT_HD (1 << 20)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
@@ -1718,10 +1725,17 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain,
* this substream's traffic
*/
} else { /* (1) and (2) */
+ u64 tcr = cd->tcr;
+
cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
cdptr[2] = 0;
cdptr[3] = cpu_to_le64(cd->mair);
+ if (!(smmu->features & ARM_SMMU_FEAT_HD))
+ tcr &= ~CTXDESC_CD_0_TCR_HD;
+ if (!(smmu->features & ARM_SMMU_FEAT_HA))
+ tcr &= ~CTXDESC_CD_0_TCR_HA;
+
/*
* STE is live, and the SMMU might read dwords of this CD in any
* order. Ensure that it observes valid values before reading
@@ -1729,7 +1743,7 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain,
*/
arm_smmu_sync_cd(smmu_domain, ssid, true);
- val = cd->tcr |
+ val = tcr |
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
#endif
@@ -1958,10 +1972,12 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
return old_cd;
}
+ /* HA and HD will be filtered out later if not supported by the SMMU */
tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) |
FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) |
FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) |
FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) |
+ CTXDESC_CD_0_TCR_HA | CTXDESC_CD_0_TCR_HD |
CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
switch (PAGE_SIZE) {
@@ -4454,6 +4470,12 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
smmu->features |= ARM_SMMU_FEAT_E2H;
}
+ if (reg & (IDR0_HA | IDR0_HD)) {
+ smmu->features |= ARM_SMMU_FEAT_HA;
+ if (reg & IDR0_HD)
+ smmu->features |= ARM_SMMU_FEAT_HD;
+ }
+
/*
* If the CPU is using VHE, but the SMMU doesn't support it, the SMMU
* will create TLB entries for NH-EL1 world and will miss the
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 14/24] iommu/arm-smmu-v3: Add SVA feature checking
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
Suzuki K Poulose, robin.murphy, hch, zhangfei.gao,
Jean-Philippe Brucker, Jonathan.Cameron, felix.kuehling, xuzaibo,
will, christian.koenig, baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Aggregate all sanity-checks for sharing CPU page tables with the SMMU
under a single ARM_SMMU_FEAT_SVA bit. For PCIe SVA, users also need to
check FEAT_ATS and FEAT_PRI. For platform SVA, they will most likely have
to check FEAT_STALLS.
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/arm-smmu-v3.c | 72 +++++++++++++++++++++++++++++++++++++
1 file changed, 72 insertions(+)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 9332253e3608..a9f6f1d7014e 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -660,6 +660,7 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
#define ARM_SMMU_FEAT_E2H (1 << 16)
#define ARM_SMMU_FEAT_BTM (1 << 17)
+#define ARM_SMMU_FEAT_SVA (1 << 18)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
@@ -3935,6 +3936,74 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
return 0;
}
+static bool arm_smmu_supports_sva(struct arm_smmu_device *smmu)
+{
+ unsigned long reg, fld;
+ unsigned long oas;
+ unsigned long asid_bits;
+
+ u32 feat_mask = ARM_SMMU_FEAT_BTM | ARM_SMMU_FEAT_COHERENCY;
+
+ if ((smmu->features & feat_mask) != feat_mask)
+ return false;
+
+ if (!(smmu->pgsize_bitmap & PAGE_SIZE))
+ return false;
+
+ /*
+ * Get the smallest PA size of all CPUs (sanitized by cpufeature). We're
+ * not even pretending to support AArch32 here.
+ */
+ reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
+ fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT);
+ switch (fld) {
+ case 0x0:
+ oas = 32;
+ break;
+ case 0x1:
+ oas = 36;
+ break;
+ case 0x2:
+ oas = 40;
+ break;
+ case 0x3:
+ oas = 42;
+ break;
+ case 0x4:
+ oas = 44;
+ break;
+ case 0x5:
+ oas = 48;
+ break;
+ case 0x6:
+ oas = 52;
+ break;
+ default:
+ return false;
+ }
+
+ /* abort if MMU outputs addresses larger than what we support. */
+ if (smmu->oas < oas)
+ return false;
+
+ /* We can support bigger ASIDs than the CPU, but not smaller */
+ fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_ASID_SHIFT);
+ asid_bits = fld ? 16 : 8;
+ if (smmu->asid_bits < asid_bits)
+ return false;
+
+ /*
+ * See max_pinned_asids in arch/arm64/mm/context.c. The following is
+ * generally the maximum number of bindable processes.
+ */
+ if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0))
+ asid_bits--;
+ dev_dbg(smmu->dev, "%d shared contexts\n", (1 << asid_bits) -
+ num_possible_cpus() - 2);
+
+ return true;
+}
+
static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
{
u32 reg;
@@ -4147,6 +4216,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
smmu->ias = max(smmu->ias, smmu->oas);
+ if (arm_smmu_supports_sva(smmu))
+ smmu->features |= ARM_SMMU_FEAT_SVA;
+
dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n",
smmu->ias, smmu->oas, smmu->features);
return 0;
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 13/24] iommu/arm-smmu-v3: Enable broadcast TLB maintenance
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
The SMMUv3 can handle invalidation targeted at TLB entries with shared
ASIDs. If the implementation supports broadcast TLB maintenance, enable it
and keep track of it in a feature bit. The SMMU will then be affected by
inner-shareable TLB invalidations from other agents.
A major side-effect of this change is that stage-2 translation contexts
are now affected by all invalidations by VMID. VMIDs are all shared and
the only ways to prevent over-invalidation, since the stage-2 page tables
are not shared between CPU and SMMU, are to either disable BTM or allocate
different VMIDs. This patch does not address the problem.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/arm-smmu-v3.c | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 7e1933e7e35f..9332253e3608 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -56,6 +56,7 @@
#define IDR0_ASID16 (1 << 12)
#define IDR0_ATS (1 << 10)
#define IDR0_HYP (1 << 9)
+#define IDR0_BTM (1 << 5)
#define IDR0_COHACC (1 << 4)
#define IDR0_TTF GENMASK(3, 2)
#define IDR0_TTF_AARCH64 2
@@ -658,6 +659,7 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_VAX (1 << 14)
#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
#define ARM_SMMU_FEAT_E2H (1 << 16)
+#define ARM_SMMU_FEAT_BTM (1 << 17)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
@@ -3819,11 +3821,14 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
writel_relaxed(reg, smmu->base + ARM_SMMU_CR1);
/* CR2 (random crap) */
- reg = CR2_PTM | CR2_RECINVSID;
+ reg = CR2_RECINVSID;
if (smmu->features & ARM_SMMU_FEAT_E2H)
reg |= CR2_E2H;
+ if (!(smmu->features & ARM_SMMU_FEAT_BTM))
+ reg |= CR2_PTM;
+
writel_relaxed(reg, smmu->base + ARM_SMMU_CR2);
/* Stream table */
@@ -3934,6 +3939,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
{
u32 reg;
bool coherent = smmu->features & ARM_SMMU_FEAT_COHERENCY;
+ bool vhe = cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN);
/* IDR0 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR0);
@@ -3983,10 +3989,19 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
if (reg & IDR0_HYP) {
smmu->features |= ARM_SMMU_FEAT_HYP;
- if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
+ if (vhe)
smmu->features |= ARM_SMMU_FEAT_E2H;
}
+ /*
+ * If the CPU is using VHE, but the SMMU doesn't support it, the SMMU
+ * will create TLB entries for NH-EL1 world and will miss the
+ * broadcasted TLB invalidations that target EL2-E2H world. Don't enable
+ * BTM in that case.
+ */
+ if (reg & IDR0_BTM && (!vhe || reg & IDR0_HYP))
+ smmu->features |= ARM_SMMU_FEAT_BTM;
+
/*
* The coherency feature as set by FW is used in preference to the ID
* register, but warn on mismatch.
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 12/24] iommu/arm-smmu-v3: Add support for VHE
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
ARMv8.1 extensions added Virtualization Host Extensions (VHE), which allow
to run a host kernel at EL2. When using normal DMA, Device and CPU address
spaces are dissociated, and do not need to implement the same
capabilities, so VHE hasn't been used in the SMMU until now.
With shared address spaces however, ASIDs are shared between MMU and SMMU,
and broadcast TLB invalidations issued by a CPU are taken into account by
the SMMU. TLB entries on both sides need to have identical exception level
in order to be cleared with a single invalidation.
When the CPU is using VHE, enable VHE in the SMMU for all STEs. Normal DMA
mappings will need to use TLBI_EL2 commands instead of TLBI_NH, but
shouldn't be otherwise affected by this change.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/arm-smmu-v3.c | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 403871d36438..7e1933e7e35f 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -13,6 +13,7 @@
#include <linux/acpi_iort.h>
#include <linux/bitfield.h>
#include <linux/bitops.h>
+#include <linux/cpufeature.h>
#include <linux/crash_dump.h>
#include <linux/delay.h>
#include <linux/dma-iommu.h>
@@ -482,6 +483,8 @@ struct arm_smmu_cmdq_ent {
#define CMDQ_OP_TLBI_NH_ASID 0x11
#define CMDQ_OP_TLBI_NH_VA 0x12
#define CMDQ_OP_TLBI_EL2_ALL 0x20
+ #define CMDQ_OP_TLBI_EL2_ASID 0x21
+ #define CMDQ_OP_TLBI_EL2_VA 0x22
#define CMDQ_OP_TLBI_S12_VMALL 0x28
#define CMDQ_OP_TLBI_S2_IPA 0x2a
#define CMDQ_OP_TLBI_NSNH_ALL 0x30
@@ -654,6 +657,7 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_STALL_FORCE (1 << 13)
#define ARM_SMMU_FEAT_VAX (1 << 14)
#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
+#define ARM_SMMU_FEAT_E2H (1 << 16)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
@@ -927,6 +931,8 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num);
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale);
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
+ /* Fallthrough */
+ case CMDQ_OP_TLBI_EL2_VA:
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf);
cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl);
@@ -948,6 +954,9 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
case CMDQ_OP_TLBI_S12_VMALL:
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
break;
+ case CMDQ_OP_TLBI_EL2_ASID:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
+ break;
case CMDQ_OP_ATC_INV:
cmd[0] |= FIELD_PREP(CMDQ_0_SSV, ent->substream_valid);
cmd[0] |= FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global);
@@ -1541,7 +1550,8 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
static void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
{
struct arm_smmu_cmdq_ent cmd = {
- .opcode = CMDQ_OP_TLBI_NH_ASID,
+ .opcode = smmu->features & ARM_SMMU_FEAT_E2H ?
+ CMDQ_OP_TLBI_EL2_ASID : CMDQ_OP_TLBI_NH_ASID,
.tlbi.asid = asid,
};
@@ -2084,13 +2094,16 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
}
if (s1_cfg) {
+ int strw = smmu->features & ARM_SMMU_FEAT_E2H ?
+ STRTAB_STE_1_STRW_EL2 : STRTAB_STE_1_STRW_NSEL1;
+
BUG_ON(ste_live);
dst[1] = cpu_to_le64(
FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
- FIELD_PREP(STRTAB_STE_1_STRW, STRTAB_STE_1_STRW_NSEL1));
+ FIELD_PREP(STRTAB_STE_1_STRW, strw));
if (smmu->features & ARM_SMMU_FEAT_STALLS &&
!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
@@ -2486,7 +2499,8 @@ static void arm_smmu_tlb_inv_range(unsigned long iova, size_t size,
return;
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
- cmd.opcode = CMDQ_OP_TLBI_NH_VA;
+ cmd.opcode = smmu->features & ARM_SMMU_FEAT_E2H ?
+ CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA;
cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid;
} else {
cmd.opcode = CMDQ_OP_TLBI_S2_IPA;
@@ -3805,7 +3819,11 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
writel_relaxed(reg, smmu->base + ARM_SMMU_CR1);
/* CR2 (random crap) */
- reg = CR2_PTM | CR2_RECINVSID | CR2_E2H;
+ reg = CR2_PTM | CR2_RECINVSID;
+
+ if (smmu->features & ARM_SMMU_FEAT_E2H)
+ reg |= CR2_E2H;
+
writel_relaxed(reg, smmu->base + ARM_SMMU_CR2);
/* Stream table */
@@ -3963,8 +3981,11 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
if (reg & IDR0_MSI)
smmu->features |= ARM_SMMU_FEAT_MSI;
- if (reg & IDR0_HYP)
+ if (reg & IDR0_HYP) {
smmu->features |= ARM_SMMU_FEAT_HYP;
+ if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
+ smmu->features |= ARM_SMMU_FEAT_E2H;
+ }
/*
* The coherency feature as set by FW is used in preference to the ID
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 08/24] iommu/arm-smmu-v3: Manage ASIDs with xarray
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
In preparation for sharing some ASIDs with the CPU, use a global xarray to
store ASIDs and their context. ASID#0 is now reserved, and the ASID
space is global.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++---------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index b5467e3e9250..847c7de0a93f 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -667,7 +667,6 @@ struct arm_smmu_device {
#define ARM_SMMU_MAX_ASIDS (1 << 16)
unsigned int asid_bits;
- DECLARE_BITMAP(asid_map, ARM_SMMU_MAX_ASIDS);
#define ARM_SMMU_MAX_VMIDS (1 << 16)
unsigned int vmid_bits;
@@ -727,6 +726,8 @@ struct arm_smmu_option_prop {
const char *prop;
};
+static DEFINE_XARRAY_ALLOC1(asid_xa);
+
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
{ ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
@@ -1765,6 +1766,14 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain)
cdcfg->cdtab = NULL;
}
+static void arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
+{
+ if (!cd->asid)
+ return;
+
+ xa_erase(&asid_xa, cd->asid);
+}
+
/* Stream table manipulation functions */
static void
arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
@@ -2450,10 +2459,9 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg;
- if (cfg->cdcfg.cdtab) {
+ if (cfg->cdcfg.cdtab)
arm_smmu_free_cd_tables(smmu_domain);
- arm_smmu_bitmap_free(smmu->asid_map, cfg->cd.asid);
- }
+ arm_smmu_free_asid(&cfg->cd);
} else {
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
if (cfg->vmid)
@@ -2468,14 +2476,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain,
struct io_pgtable_cfg *pgtbl_cfg)
{
int ret;
- int asid;
+ u32 asid;
struct arm_smmu_device *smmu = smmu_domain->smmu;
struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg;
typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
- asid = arm_smmu_bitmap_alloc(smmu->asid_map, smmu->asid_bits);
- if (asid < 0)
- return asid;
+ ret = xa_alloc(&asid_xa, &asid, &cfg->cd,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ if (ret)
+ return ret;
cfg->s1cdmax = master->ssid_bits;
@@ -2508,7 +2517,7 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain,
out_free_cd_tables:
arm_smmu_free_cd_tables(smmu_domain);
out_free_asid:
- arm_smmu_bitmap_free(smmu->asid_map, asid);
+ arm_smmu_free_asid(&cfg->cd);
return ret;
}
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 10/24] iommu/arm-smmu-v3: Share process page tables
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
Suzuki K Poulose, robin.murphy, hch, zhangfei.gao,
Jean-Philippe Brucker, Jonathan.Cameron, felix.kuehling, xuzaibo,
will, christian.koenig, baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
With Shared Virtual Addressing (SVA), we need to mirror CPU TTBR, TCR,
MAIR and ASIDs in SMMU contexts. Each SMMU has a single ASID space split
into two sets, shared and private. Shared ASIDs correspond to those
obtained from the arch ASID allocator, and private ASIDs are used for
"classic" map/unmap DMA.
Each mm_struct shared with the SMMU will have a single context
descriptor. Add a refcount to keep track of this. It will be protected
by the global SVA lock.
Acked-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
v6->v7: Add lockdep annotations for sva_lock
---
drivers/iommu/arm-smmu-v3.c | 153 +++++++++++++++++++++++++++++++++++-
1 file changed, 149 insertions(+), 4 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 847c7de0a93f..52cbdf08f5e2 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -22,6 +22,7 @@
#include <linux/iommu.h>
#include <linux/iopoll.h>
#include <linux/module.h>
+#include <linux/mmu_context.h>
#include <linux/msi.h>
#include <linux/of.h>
#include <linux/of_address.h>
@@ -33,6 +34,8 @@
#include <linux/amba/bus.h>
+#include "io-pgtable-arm.h"
+
/* MMIO registers */
#define ARM_SMMU_IDR0 0x0
#define IDR0_ST_LVL GENMASK(28, 27)
@@ -589,6 +592,9 @@ struct arm_smmu_ctx_desc {
u64 ttbr;
u64 tcr;
u64 mair;
+
+ refcount_t refs;
+ struct mm_struct *mm;
};
struct arm_smmu_l1_ctx_desc {
@@ -727,6 +733,7 @@ struct arm_smmu_option_prop {
};
static DEFINE_XARRAY_ALLOC1(asid_xa);
+static DEFINE_MUTEX(sva_lock);
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
@@ -1662,7 +1669,8 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain,
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
#endif
- CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET |
+ CTXDESC_CD_0_R | CTXDESC_CD_0_A |
+ (cd->mm ? 0 : CTXDESC_CD_0_ASET) |
CTXDESC_CD_0_AA64 |
FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) |
CTXDESC_CD_0_V;
@@ -1766,12 +1774,147 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain)
cdcfg->cdtab = NULL;
}
-static void arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
+static void arm_smmu_init_cd(struct arm_smmu_ctx_desc *cd)
{
+ refcount_set(&cd->refs, 1);
+}
+
+static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
+{
+ bool free;
+ struct arm_smmu_ctx_desc *old_cd;
+
if (!cd->asid)
- return;
+ return false;
+
+ free = refcount_dec_and_test(&cd->refs);
+ if (free) {
+ old_cd = xa_erase(&asid_xa, cd->asid);
+ WARN_ON(old_cd != cd);
+ }
+ return free;
+}
+
+static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid)
+{
+ struct arm_smmu_ctx_desc *cd;
+
+ cd = xa_load(&asid_xa, asid);
+ if (!cd)
+ return NULL;
+
+ if (cd->mm) {
+ /* All devices bound to this mm use the same cd struct. */
+ refcount_inc(&cd->refs);
+ return cd;
+ }
+
+ /*
+ * Ouch, ASID is already in use for a private cd.
+ * TODO: seize it.
+ */
+ return ERR_PTR(-EEXIST);
+}
+
+__maybe_unused
+static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
+{
+ u16 asid;
+ int ret = 0;
+ u64 tcr, par, reg;
+ struct arm_smmu_ctx_desc *cd;
+ struct arm_smmu_ctx_desc *old_cd = NULL;
+
+ lockdep_assert_held(&sva_lock);
+
+ asid = mm_context_get(mm);
+ if (!asid)
+ return ERR_PTR(-ESRCH);
- xa_erase(&asid_xa, cd->asid);
+ cd = kzalloc(sizeof(*cd), GFP_KERNEL);
+ if (!cd) {
+ ret = -ENOMEM;
+ goto err_put_context;
+ }
+
+ arm_smmu_init_cd(cd);
+
+ old_cd = arm_smmu_share_asid(asid);
+ if (IS_ERR(old_cd)) {
+ ret = PTR_ERR(old_cd);
+ goto err_free_cd;
+ } else if (old_cd) {
+ if (WARN_ON(old_cd->mm != mm)) {
+ ret = -EINVAL;
+ goto err_free_cd;
+ }
+ kfree(cd);
+ mm_context_put(mm);
+ return old_cd;
+ }
+
+ /* Fails if a private ASID has been allocated since we last checked */
+ ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL);
+ if (ret)
+ goto err_free_cd;
+
+ tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) |
+ CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
+
+ switch (PAGE_SIZE) {
+ case SZ_4K:
+ tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K);
+ break;
+ case SZ_16K:
+ tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K);
+ break;
+ case SZ_64K:
+ tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K);
+ break;
+ default:
+ WARN_ON(1);
+ ret = -EINVAL;
+ goto err_free_asid;
+ }
+
+ reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
+ par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT);
+ tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par);
+
+ cd->ttbr = virt_to_phys(mm->pgd);
+ cd->tcr = tcr;
+ /*
+ * MAIR value is pretty much constant and global, so we can just get it
+ * from the current CPU register
+ */
+ cd->mair = read_sysreg(mair_el1);
+ cd->asid = asid;
+ cd->mm = mm;
+
+ return cd;
+
+err_free_asid:
+ arm_smmu_free_asid(cd);
+err_free_cd:
+ kfree(cd);
+err_put_context:
+ mm_context_put(mm);
+ return ERR_PTR(ret);
+}
+
+__maybe_unused
+static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd)
+{
+ lockdep_assert_held(&sva_lock);
+
+ if (arm_smmu_free_asid(cd)) {
+ /* Unpin ASID */
+ mm_context_put(cd->mm);
+ kfree(cd);
+ }
}
/* Stream table manipulation functions */
@@ -2481,6 +2624,8 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain,
struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg;
typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
+ arm_smmu_init_cd(&cfg->cd);
+
ret = xa_alloc(&asid_xa, &asid, &cfg->cd,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
if (ret)
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 11/24] iommu/arm-smmu-v3: Seize private ASID
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
The SMMU has a single ASID space, the union of shared and private ASID
sets. This means that the SMMU driver competes with the arch allocator
for ASIDs. Shared ASIDs are those of Linux processes, allocated by the
arch, and contribute in broadcast TLB maintenance. Private ASIDs are
allocated by the SMMU driver and used for "classic" map/unmap DMA. They
require command-queue TLB invalidations.
When we pin down an mm_context and get an ASID that is already in use by
the SMMU, it belongs to a private context. We used to simply abort the
bind, but this is unfair to users that would be unable to bind a few
seemingly random processes. Try to allocate a new private ASID for the
context, and make the old ASID shared.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
v6->v7: Replace context_lock spinlock with asid_lock mutex, remove
GFP_ATOMIC changes, add comments about locking.
---
drivers/iommu/arm-smmu-v3.c | 100 ++++++++++++++++++++++++++++--------
1 file changed, 80 insertions(+), 20 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 52cbdf08f5e2..403871d36438 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -733,6 +733,7 @@ struct arm_smmu_option_prop {
};
static DEFINE_XARRAY_ALLOC1(asid_xa);
+static DEFINE_MUTEX(asid_lock);
static DEFINE_MUTEX(sva_lock);
static struct arm_smmu_option_prop arm_smmu_options[] = {
@@ -1537,6 +1538,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
}
/* Context descriptor manipulation functions */
+static void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
+{
+ struct arm_smmu_cmdq_ent cmd = {
+ .opcode = CMDQ_OP_TLBI_NH_ASID,
+ .tlbi.asid = asid,
+ };
+
+ arm_smmu_cmdq_issue_cmd(smmu, &cmd);
+ arm_smmu_cmdq_issue_sync(smmu);
+}
+
static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain,
int ssid, bool leaf)
{
@@ -1795,9 +1807,18 @@ static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
return free;
}
+/*
+ * Try to reserve this ASID in the SMMU. If it is in use, try to steal it from
+ * the private entry. Careful here, we may be modifying the context tables of
+ * another SMMU!
+ */
static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid)
{
+ int ret;
+ u32 new_asid;
struct arm_smmu_ctx_desc *cd;
+ struct arm_smmu_device *smmu;
+ struct arm_smmu_domain *smmu_domain;
cd = xa_load(&asid_xa, asid);
if (!cd)
@@ -1809,11 +1830,31 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid)
return cd;
}
+ smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd);
+ smmu = smmu_domain->smmu;
+
+ ret = xa_alloc(&asid_xa, &new_asid, cd,
+ XA_LIMIT(1, 1 << smmu->asid_bits), GFP_KERNEL);
+ if (ret)
+ return ERR_PTR(-ENOSPC);
+ /*
+ * Race with unmap: TLB invalidations will start targeting the new ASID,
+ * which isn't assigned yet. We'll do an invalidate-all on the old ASID
+ * later, so it doesn't matter.
+ */
+ cd->asid = new_asid;
+
/*
- * Ouch, ASID is already in use for a private cd.
- * TODO: seize it.
+ * Update ASID and invalidate CD in all associated masters. There will
+ * be some overlap between use of both ASIDs, until we invalidate the
+ * TLB.
*/
- return ERR_PTR(-EEXIST);
+ arm_smmu_write_ctx_desc(smmu_domain, 0, cd);
+
+ /* Invalidate TLB entries previously associated with that context */
+ arm_smmu_tlb_inv_asid(smmu, asid);
+
+ return NULL;
}
__maybe_unused
@@ -1839,7 +1880,20 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
arm_smmu_init_cd(cd);
+ /*
+ * Serialize against arm_smmu_domain_finalise_s1() and
+ * arm_smmu_domain_free() as we might need to replace the private ASID
+ * from an existing CD.
+ */
+ mutex_lock(&asid_lock);
old_cd = arm_smmu_share_asid(asid);
+ if (!old_cd) {
+ ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL);
+ if (ret)
+ old_cd = ERR_PTR(ret);
+ }
+ mutex_unlock(&asid_lock);
+
if (IS_ERR(old_cd)) {
ret = PTR_ERR(old_cd);
goto err_free_cd;
@@ -1853,11 +1907,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
return old_cd;
}
- /* Fails if a private ASID has been allocated since we last checked */
- ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL);
- if (ret)
- goto err_free_cd;
-
tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) |
FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) |
FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) |
@@ -2401,15 +2450,6 @@ static void arm_smmu_tlb_inv_context(void *cookie)
struct arm_smmu_device *smmu = smmu_domain->smmu;
struct arm_smmu_cmdq_ent cmd;
- if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
- cmd.opcode = CMDQ_OP_TLBI_NH_ASID;
- cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid;
- cmd.tlbi.vmid = 0;
- } else {
- cmd.opcode = CMDQ_OP_TLBI_S12_VMALL;
- cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
- }
-
/*
* NOTE: when io-pgtable is in non-strict mode, we may get here with
* PTEs previously cleared by unmaps on the current CPU not yet visible
@@ -2417,8 +2457,14 @@ static void arm_smmu_tlb_inv_context(void *cookie)
* insertion to guarantee those are observed before the TLBI. Do be
* careful, 007.
*/
- arm_smmu_cmdq_issue_cmd(smmu, &cmd);
- arm_smmu_cmdq_issue_sync(smmu);
+ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
+ arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid);
+ } else {
+ cmd.opcode = CMDQ_OP_TLBI_S12_VMALL;
+ cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
+ arm_smmu_cmdq_issue_cmd(smmu, &cmd);
+ arm_smmu_cmdq_issue_sync(smmu);
+ }
arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0);
}
@@ -2602,9 +2648,15 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg;
+ /*
+ * Prevent arm_smmu_share_asid() from rewriting CD#0 while we're
+ * freeing it.
+ */
+ mutex_lock(&asid_lock);
if (cfg->cdcfg.cdtab)
arm_smmu_free_cd_tables(smmu_domain);
arm_smmu_free_asid(&cfg->cd);
+ mutex_unlock(&asid_lock);
} else {
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
if (cfg->vmid)
@@ -2626,10 +2678,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain,
arm_smmu_init_cd(&cfg->cd);
+ /*
+ * Prevent arm_smmu_share_asid() from seizing the private ASID we're
+ * allocating here until it is written to the CD.
+ */
+ mutex_lock(&asid_lock);
ret = xa_alloc(&asid_xa, &asid, &cfg->cd,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
if (ret)
- return ret;
+ goto out_unlock;
cfg->s1cdmax = master->ssid_bits;
@@ -2657,12 +2714,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain,
if (ret)
goto out_free_cd_tables;
+ mutex_unlock(&asid_lock);
return 0;
out_free_cd_tables:
arm_smmu_free_cd_tables(smmu_domain);
out_free_asid:
arm_smmu_free_asid(&cfg->cd);
+out_unlock:
+ mutex_unlock(&asid_lock);
return ret;
}
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 09/24] arm64: cpufeature: Export symbol read_sanitised_ftr_reg()
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
Suzuki K Poulose, robin.murphy, hch, zhangfei.gao,
Jean-Philippe Brucker, Jonathan.Cameron, felix.kuehling, xuzaibo,
will, christian.koenig, baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
The SMMUv3 driver would like to read the MMFR0 PARANGE field in order to
share CPU page tables with devices. Allow the driver to be built as
module by exporting the read_sanitized_ftr_reg() cpufeature symbol.
Acked-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
arch/arm64/kernel/cpufeature.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9fac745aa7bb..5f6adbf4ae89 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -841,6 +841,7 @@ u64 read_sanitised_ftr_reg(u32 id)
BUG_ON(!regp);
return regp->sys_val;
}
+EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg);
#define read_sysreg_case(r) \
case r: return read_sysreg_s(r)
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 07/24] iommu/io-pgtable-arm: Move some definitions to a header
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Extract some of the most generic TCR defines, so they can be reused by
the page table sharing code.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/io-pgtable-arm.h | 30 ++++++++++++++++++++++++++++++
drivers/iommu/io-pgtable-arm.c | 27 ++-------------------------
MAINTAINERS | 3 +--
3 files changed, 33 insertions(+), 27 deletions(-)
create mode 100644 drivers/iommu/io-pgtable-arm.h
diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h
new file mode 100644
index 000000000000..ba7cfdf7afa0
--- /dev/null
+++ b/drivers/iommu/io-pgtable-arm.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef IO_PGTABLE_ARM_H_
+#define IO_PGTABLE_ARM_H_
+
+#define ARM_LPAE_TCR_TG0_4K 0
+#define ARM_LPAE_TCR_TG0_64K 1
+#define ARM_LPAE_TCR_TG0_16K 2
+
+#define ARM_LPAE_TCR_TG1_16K 1
+#define ARM_LPAE_TCR_TG1_4K 2
+#define ARM_LPAE_TCR_TG1_64K 3
+
+#define ARM_LPAE_TCR_SH_NS 0
+#define ARM_LPAE_TCR_SH_OS 2
+#define ARM_LPAE_TCR_SH_IS 3
+
+#define ARM_LPAE_TCR_RGN_NC 0
+#define ARM_LPAE_TCR_RGN_WBWA 1
+#define ARM_LPAE_TCR_RGN_WT 2
+#define ARM_LPAE_TCR_RGN_WB 3
+
+#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL
+#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL
+#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL
+#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL
+#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL
+#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL
+#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL
+
+#endif /* IO_PGTABLE_ARM_H_ */
diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
index 04fbd4bf0ff9..f71a2eade04a 100644
--- a/drivers/iommu/io-pgtable-arm.c
+++ b/drivers/iommu/io-pgtable-arm.c
@@ -20,6 +20,8 @@
#include <asm/barrier.h>
+#include "io-pgtable-arm.h"
+
#define ARM_LPAE_MAX_ADDR_BITS 52
#define ARM_LPAE_S2_MAX_CONCAT_PAGES 16
#define ARM_LPAE_MAX_LEVELS 4
@@ -100,23 +102,6 @@
#define ARM_LPAE_PTE_MEMATTR_DEV (((arm_lpae_iopte)0x1) << 2)
/* Register bits */
-#define ARM_LPAE_TCR_TG0_4K 0
-#define ARM_LPAE_TCR_TG0_64K 1
-#define ARM_LPAE_TCR_TG0_16K 2
-
-#define ARM_LPAE_TCR_TG1_16K 1
-#define ARM_LPAE_TCR_TG1_4K 2
-#define ARM_LPAE_TCR_TG1_64K 3
-
-#define ARM_LPAE_TCR_SH_NS 0
-#define ARM_LPAE_TCR_SH_OS 2
-#define ARM_LPAE_TCR_SH_IS 3
-
-#define ARM_LPAE_TCR_RGN_NC 0
-#define ARM_LPAE_TCR_RGN_WBWA 1
-#define ARM_LPAE_TCR_RGN_WT 2
-#define ARM_LPAE_TCR_RGN_WB 3
-
#define ARM_LPAE_VTCR_SL0_MASK 0x3
#define ARM_LPAE_TCR_T0SZ_SHIFT 0
@@ -124,14 +109,6 @@
#define ARM_LPAE_VTCR_PS_SHIFT 16
#define ARM_LPAE_VTCR_PS_MASK 0x7
-#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL
-#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL
-#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL
-#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL
-#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL
-#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL
-#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL
-
#define ARM_LPAE_MAIR_ATTR_SHIFT(n) ((n) << 3)
#define ARM_LPAE_MAIR_ATTR_MASK 0xff
#define ARM_LPAE_MAIR_ATTR_DEVICE 0x04
diff --git a/MAINTAINERS b/MAINTAINERS
index ecc0749810b0..4ff7b9a5bb7d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1463,8 +1463,7 @@ L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
S: Maintained
F: Documentation/devicetree/bindings/iommu/arm,smmu*
F: drivers/iommu/arm-smmu*
-F: drivers/iommu/io-pgtable-arm-v7s.c
-F: drivers/iommu/io-pgtable-arm.c
+F: drivers/iommu/io-pgtable-arm*
ARM SUB-ARCHITECTURES
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 06/24] arm64: mm: Pin down ASIDs for sharing mm with devices
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
To enable address space sharing with the IOMMU, introduce mm_context_get()
and mm_context_put(), that pin down a context and ensure that it will keep
its ASID after a rollover. Export the symbols to let the modular SMMUv3
driver use them.
Pinning is necessary because a device constantly needs a valid ASID,
unlike tasks that only require one when running. Without pinning, we would
need to notify the IOMMU when we're about to use a new ASID for a task,
and it would get complicated when a new task is assigned a shared ASID.
Consider the following scenario with no ASID pinned:
1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1)
2. Task t2 is scheduled on CPUx, gets ASID (1, 2)
3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1)
We would now have to immediately generate a new ASID for t1, notify
the IOMMU, and finally enable task tn. We are holding the lock during
all that time, since we can't afford having another CPU trigger a
rollover. The IOMMU issues invalidation commands that can take tens of
milliseconds.
It gets needlessly complicated. All we wanted to do was schedule task tn,
that has no business with the IOMMU. By letting the IOMMU pin tasks when
needed, we avoid stalling the slow path, and let the pinning fail when
we're out of shareable ASIDs.
After a rollover, the allocator expects at least one ASID to be available
in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS -
1) is the maximum number of ASIDs that can be shared with the IOMMU.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
arch/arm64/include/asm/mmu.h | 1 +
arch/arm64/include/asm/mmu_context.h | 11 +++-
arch/arm64/mm/context.c | 95 +++++++++++++++++++++++++++-
3 files changed, 104 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
index 68140fdd89d6..bbdd291e31d5 100644
--- a/arch/arm64/include/asm/mmu.h
+++ b/arch/arm64/include/asm/mmu.h
@@ -19,6 +19,7 @@
typedef struct {
atomic64_t id;
+ unsigned long pinned;
void *vdso;
unsigned long flags;
} mm_context_t;
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index ab46187c6300..69599a64945b 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -177,7 +177,13 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp)
#define destroy_context(mm) do { } while(0)
void check_and_switch_context(struct mm_struct *mm, unsigned int cpu);
-#define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; })
+static inline int
+init_new_context(struct task_struct *tsk, struct mm_struct *mm)
+{
+ atomic64_set(&mm->context.id, 0);
+ mm->context.pinned = 0;
+ return 0;
+}
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
static inline void update_saved_ttbr0(struct task_struct *tsk,
@@ -250,6 +256,9 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
void verify_cpu_asid_bits(void);
void post_ttbr_update_workaround(void);
+unsigned long mm_context_get(struct mm_struct *mm);
+void mm_context_put(struct mm_struct *mm);
+
#endif /* !__ASSEMBLY__ */
#endif /* !__ASM_MMU_CONTEXT_H */
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index d702d60e64da..d0ddd413f564 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -27,6 +27,10 @@ static DEFINE_PER_CPU(atomic64_t, active_asids);
static DEFINE_PER_CPU(u64, reserved_asids);
static cpumask_t tlb_flush_pending;
+static unsigned long max_pinned_asids;
+static unsigned long nr_pinned_asids;
+static unsigned long *pinned_asid_map;
+
#define ASID_MASK (~GENMASK(asid_bits - 1, 0))
#define ASID_FIRST_VERSION (1UL << asid_bits)
@@ -74,6 +78,9 @@ void verify_cpu_asid_bits(void)
static void set_kpti_asid_bits(void)
{
+ unsigned int k;
+ u8 *dst = (u8 *)asid_map;
+ u8 *src = (u8 *)pinned_asid_map;
unsigned int len = BITS_TO_LONGS(NUM_USER_ASIDS) * sizeof(unsigned long);
/*
* In case of KPTI kernel/user ASIDs are allocated in
@@ -81,7 +88,8 @@ static void set_kpti_asid_bits(void)
* is set, then the ASID will map only userspace. Thus
* mark even as reserved for kernel.
*/
- memset(asid_map, 0xaa, len);
+ for (k = 0; k < len; k++)
+ dst[k] = src[k] | 0xaa;
}
static void set_reserved_asid_bits(void)
@@ -89,7 +97,7 @@ static void set_reserved_asid_bits(void)
if (arm64_kernel_unmapped_at_el0())
set_kpti_asid_bits();
else
- bitmap_clear(asid_map, 0, NUM_USER_ASIDS);
+ bitmap_copy(asid_map, pinned_asid_map, NUM_USER_ASIDS);
}
#define asid_gen_match(asid) \
@@ -165,6 +173,14 @@ static u64 new_context(struct mm_struct *mm)
if (check_update_reserved_asid(asid, newasid))
return newasid;
+ /*
+ * If it is pinned, we can keep using it. Note that reserved
+ * takes priority, because even if it is also pinned, we need to
+ * update the generation into the reserved_asids.
+ */
+ if (mm->context.pinned)
+ return newasid;
+
/*
* We had a valid ASID in a previous life, so try to re-use
* it if possible.
@@ -254,6 +270,68 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu)
cpu_switch_mm(mm->pgd, mm);
}
+unsigned long mm_context_get(struct mm_struct *mm)
+{
+ unsigned long flags;
+ u64 asid;
+
+ raw_spin_lock_irqsave(&cpu_asid_lock, flags);
+
+ asid = atomic64_read(&mm->context.id);
+
+ if (mm->context.pinned) {
+ mm->context.pinned++;
+ asid &= ~ASID_MASK;
+ goto out_unlock;
+ }
+
+ if (nr_pinned_asids >= max_pinned_asids) {
+ asid = 0;
+ goto out_unlock;
+ }
+
+ if (!asid_gen_match(asid)) {
+ /*
+ * We went through one or more rollover since that ASID was
+ * used. Ensure that it is still valid, or generate a new one.
+ */
+ asid = new_context(mm);
+ atomic64_set(&mm->context.id, asid);
+ }
+
+ asid &= ~ASID_MASK;
+
+ nr_pinned_asids++;
+ __set_bit(asid2idx(asid), pinned_asid_map);
+ mm->context.pinned++;
+
+out_unlock:
+ raw_spin_unlock_irqrestore(&cpu_asid_lock, flags);
+
+ /* Set the equivalent of USER_ASID_BIT */
+ if (asid && IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0))
+ asid |= 1;
+
+ return asid;
+}
+EXPORT_SYMBOL_GPL(mm_context_get);
+
+void mm_context_put(struct mm_struct *mm)
+{
+ unsigned long flags;
+ u64 asid = atomic64_read(&mm->context.id) & ~ASID_MASK;
+
+ raw_spin_lock_irqsave(&cpu_asid_lock, flags);
+
+ if (--mm->context.pinned == 0) {
+ __clear_bit(asid2idx(asid), pinned_asid_map);
+ nr_pinned_asids--;
+ }
+
+ raw_spin_unlock_irqrestore(&cpu_asid_lock, flags);
+}
+EXPORT_SYMBOL_GPL(mm_context_put);
+
/* Errata workaround post TTBRx_EL1 update. */
asmlinkage void post_ttbr_update_workaround(void)
{
@@ -303,6 +381,13 @@ static int asids_update_limit(void)
WARN_ON(num_available_asids - 1 <= num_possible_cpus());
pr_info("ASID allocator initialised with %lu entries\n",
num_available_asids);
+
+ /*
+ * We assume that an ASID is always available after a rollover. This
+ * means that even if all CPUs have a reserved ASID, there still is at
+ * least one slot available in the asid map.
+ */
+ max_pinned_asids = num_available_asids - num_possible_cpus() - 2;
return 0;
}
arch_initcall(asids_update_limit);
@@ -317,6 +402,12 @@ static int asids_init(void)
panic("Failed to allocate bitmap for %lu ASIDs\n",
NUM_USER_ASIDS);
+ pinned_asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS),
+ sizeof(*pinned_asid_map), GFP_KERNEL);
+ if (!pinned_asid_map)
+ panic("Failed to allocate pinned ASID bitmap\n");
+ nr_pinned_asids = 0;
+
/*
* We cannot call set_reserved_asid_bits() here because CPU
* caps are not finalized yet, so it is safer to assume KPTI
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 04/24] iommu: Add a page fault handler
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Some systems allow devices to handle I/O Page Faults in the core mm. For
example systems implementing the PCIe PRI extension or Arm SMMU stall
model. Infrastructure for reporting these recoverable page faults was
added to the IOMMU core by commit 0c830e6b3282 ("iommu: Introduce device
fault report API"). Add a page fault handler for host SVA.
IOMMU driver can now instantiate several fault workqueues and link them
to IOPF-capable devices. Drivers can choose between a single global
workqueue, one per IOMMU device, one per low-level fault queue, one per
domain, etc.
When it receives a fault event, supposedly in an IRQ handler, the IOMMU
driver reports the fault using iommu_report_device_fault(), which calls
the registered handler. The page fault handler then calls the mm fault
handler, and reports either success or failure with iommu_page_response().
When the handler succeeded, the IOMMU retries the access.
The iopf_param pointer could be embedded into iommu_fault_param. But
putting iopf_param into the iommu_param structure allows us not to care
about ordering between calls to iopf_queue_add_device() and
iommu_register_device_fault_handler().
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
v6->v7: Fix leak in iopf_queue_discard_partial()
---
drivers/iommu/Kconfig | 4 +
drivers/iommu/Makefile | 1 +
include/linux/iommu.h | 51 +++++
drivers/iommu/io-pgfault.c | 459 +++++++++++++++++++++++++++++++++++++
4 files changed, 515 insertions(+)
create mode 100644 drivers/iommu/io-pgfault.c
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index d9fa5b410015..15e9dc4e503c 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -107,6 +107,10 @@ config IOMMU_SVA
bool
select IOASID
+config IOMMU_PAGE_FAULT
+ bool
+ select IOMMU_SVA
+
config FSL_PAMU
bool "Freescale IOMMU support"
depends on PCI
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 40c800dd4e3e..bf5cb4ee8409 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -4,6 +4,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_IOMMU_PAGE_FAULT) += io-pgfault.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index b62525747bd9..a462157c855b 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -46,6 +46,7 @@ struct iommu_domain;
struct notifier_block;
struct iommu_sva;
struct iommu_fault_event;
+struct iopf_queue;
/* iommu fault flags */
#define IOMMU_FAULT_READ 0x0
@@ -347,6 +348,7 @@ struct iommu_fault_param {
* struct dev_iommu - Collection of per-device IOMMU data
*
* @fault_param: IOMMU detected device fault reporting data
+ * @iopf_param: I/O Page Fault queue and data
* @fwspec: IOMMU fwspec data
* @priv: IOMMU Driver private data
*
@@ -356,6 +358,7 @@ struct iommu_fault_param {
struct dev_iommu {
struct mutex lock;
struct iommu_fault_param *fault_param;
+ struct iopf_device_param *iopf_param;
struct iommu_fwspec *fwspec;
void *priv;
};
@@ -1067,4 +1070,52 @@ void iommu_debugfs_setup(void);
static inline void iommu_debugfs_setup(void) {}
#endif
+#ifdef CONFIG_IOMMU_PAGE_FAULT
+extern int iommu_queue_iopf(struct iommu_fault *fault, void *cookie);
+
+extern int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev);
+extern int iopf_queue_remove_device(struct iopf_queue *queue,
+ struct device *dev);
+extern int iopf_queue_flush_dev(struct device *dev);
+extern struct iopf_queue *iopf_queue_alloc(const char *name);
+extern void iopf_queue_free(struct iopf_queue *queue);
+extern int iopf_queue_discard_partial(struct iopf_queue *queue);
+#else /* CONFIG_IOMMU_PAGE_FAULT */
+static inline int iommu_queue_iopf(struct iommu_fault *fault, void *cookie)
+{
+ return -ENODEV;
+}
+
+static inline int iopf_queue_add_device(struct iopf_queue *queue,
+ struct device *dev)
+{
+ return -ENODEV;
+}
+
+static inline int iopf_queue_remove_device(struct iopf_queue *queue,
+ struct device *dev)
+{
+ return -ENODEV;
+}
+
+static inline int iopf_queue_flush_dev(struct device *dev)
+{
+ return -ENODEV;
+}
+
+static inline struct iopf_queue *iopf_queue_alloc(const char *name)
+{
+ return NULL;
+}
+
+static inline void iopf_queue_free(struct iopf_queue *queue)
+{
+}
+
+static inline int iopf_queue_discard_partial(struct iopf_queue *queue)
+{
+ return -ENODEV;
+}
+#endif /* CONFIG_IOMMU_PAGE_FAULT */
+
#endif /* __LINUX_IOMMU_H */
diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c
new file mode 100644
index 000000000000..1f61c1bc05da
--- /dev/null
+++ b/drivers/iommu/io-pgfault.c
@@ -0,0 +1,459 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Handle device page faults
+ *
+ * Copyright (C) 2020 ARM Ltd.
+ */
+
+#include <linux/iommu.h>
+#include <linux/list.h>
+#include <linux/sched/mm.h>
+#include <linux/slab.h>
+#include <linux/workqueue.h>
+
+#include "iommu-sva.h"
+
+/**
+ * struct iopf_queue - IO Page Fault queue
+ * @wq: the fault workqueue
+ * @devices: devices attached to this queue
+ * @lock: protects the device list
+ */
+struct iopf_queue {
+ struct workqueue_struct *wq;
+ struct list_head devices;
+ struct mutex lock;
+};
+
+/**
+ * struct iopf_device_param - IO Page Fault data attached to a device
+ * @dev: the device that owns this param
+ * @queue: IOPF queue
+ * @queue_list: index into queue->devices
+ * @partial: faults that are part of a Page Request Group for which the last
+ * request hasn't been submitted yet.
+ */
+struct iopf_device_param {
+ struct device *dev;
+ struct iopf_queue *queue;
+ struct list_head queue_list;
+ struct list_head partial;
+};
+
+struct iopf_fault {
+ struct iommu_fault fault;
+ struct list_head list;
+};
+
+struct iopf_group {
+ struct iopf_fault last_fault;
+ struct list_head faults;
+ struct work_struct work;
+ struct device *dev;
+};
+
+static int iopf_complete_group(struct device *dev, struct iopf_fault *iopf,
+ enum iommu_page_response_code status)
+{
+ struct iommu_page_response resp = {
+ .version = IOMMU_PAGE_RESP_VERSION_1,
+ .pasid = iopf->fault.prm.pasid,
+ .grpid = iopf->fault.prm.grpid,
+ .code = status,
+ };
+
+ if (iopf->fault.prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)
+ resp.flags = IOMMU_PAGE_RESP_PASID_VALID;
+
+ return iommu_page_response(dev, &resp);
+}
+
+static enum iommu_page_response_code
+iopf_handle_single(struct iopf_fault *iopf)
+{
+ vm_fault_t ret;
+ struct mm_struct *mm;
+ struct vm_area_struct *vma;
+ unsigned int access_flags = 0;
+ unsigned int fault_flags = FAULT_FLAG_REMOTE;
+ struct iommu_fault_page_request *prm = &iopf->fault.prm;
+ enum iommu_page_response_code status = IOMMU_PAGE_RESP_INVALID;
+
+ if (!(prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID))
+ return status;
+
+ mm = iommu_sva_find(prm->pasid);
+ if (IS_ERR_OR_NULL(mm))
+ return status;
+
+ down_read(&mm->mmap_sem);
+
+ vma = find_extend_vma(mm, prm->addr);
+ if (!vma)
+ /* Unmapped area */
+ goto out_put_mm;
+
+ if (prm->perm & IOMMU_FAULT_PERM_READ)
+ access_flags |= VM_READ;
+
+ if (prm->perm & IOMMU_FAULT_PERM_WRITE) {
+ access_flags |= VM_WRITE;
+ fault_flags |= FAULT_FLAG_WRITE;
+ }
+
+ if (prm->perm & IOMMU_FAULT_PERM_EXEC) {
+ access_flags |= VM_EXEC;
+ fault_flags |= FAULT_FLAG_INSTRUCTION;
+ }
+
+ if (!(prm->perm & IOMMU_FAULT_PERM_PRIV))
+ fault_flags |= FAULT_FLAG_USER;
+
+ if (access_flags & ~vma->vm_flags)
+ /* Access fault */
+ goto out_put_mm;
+
+ ret = handle_mm_fault(vma, prm->addr, fault_flags);
+ status = ret & VM_FAULT_ERROR ? IOMMU_PAGE_RESP_INVALID :
+ IOMMU_PAGE_RESP_SUCCESS;
+
+out_put_mm:
+ up_read(&mm->mmap_sem);
+ mmput(mm);
+
+ return status;
+}
+
+static void iopf_handle_group(struct work_struct *work)
+{
+ struct iopf_group *group;
+ struct iopf_fault *iopf, *next;
+ enum iommu_page_response_code status = IOMMU_PAGE_RESP_SUCCESS;
+
+ group = container_of(work, struct iopf_group, work);
+
+ list_for_each_entry_safe(iopf, next, &group->faults, list) {
+ /*
+ * For the moment, errors are sticky: don't handle subsequent
+ * faults in the group if there is an error.
+ */
+ if (status == IOMMU_PAGE_RESP_SUCCESS)
+ status = iopf_handle_single(iopf);
+
+ if (!(iopf->fault.prm.flags &
+ IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE))
+ kfree(iopf);
+ }
+
+ iopf_complete_group(group->dev, &group->last_fault, status);
+ kfree(group);
+}
+
+/**
+ * iommu_queue_iopf - IO Page Fault handler
+ * @evt: fault event
+ * @cookie: struct device, passed to iommu_register_device_fault_handler.
+ *
+ * Add a fault to the device workqueue, to be handled by mm.
+ *
+ * This module doesn't handle PCI PASID Stop Marker; IOMMU drivers must discard
+ * them before reporting faults. A PASID Stop Marker (LRW = 0b100) doesn't
+ * expect a response. It may be generated when disabling a PASID (issuing a
+ * PASID stop request) by some PCI devices.
+ *
+ * The PASID stop request is issued by the device driver before unbind(). Once
+ * it completes, no page request is generated for this PASID anymore and
+ * outstanding ones have been pushed to the IOMMU (as per PCIe 4.0r1.0 - 6.20.1
+ * and 10.4.1.2 - Managing PASID TLP Prefix Usage). Some PCI devices will wait
+ * for all outstanding page requests to come back with a response before
+ * completing the PASID stop request. Others do not wait for page responses, and
+ * instead issue this Stop Marker that tells us when the PASID can be
+ * reallocated.
+ *
+ * It is safe to discard the Stop Marker because it is an optimization.
+ * a. Page requests, which are posted requests, have been flushed to the IOMMU
+ * when the stop request completes.
+ * b. We flush all fault queues on unbind() before freeing the PASID.
+ *
+ * So even though the Stop Marker might be issued by the device *after* the stop
+ * request completes, outstanding faults will have been dealt with by the time
+ * we free the PASID.
+ *
+ * Return: 0 on success and <0 on error.
+ */
+int iommu_queue_iopf(struct iommu_fault *fault, void *cookie)
+{
+ int ret;
+ struct iopf_group *group;
+ struct iopf_fault *iopf, *next;
+ struct iopf_device_param *iopf_param;
+
+ struct device *dev = cookie;
+ struct dev_iommu *param = dev->iommu;
+
+ lockdep_assert_held(¶m->lock);
+
+ if (fault->type != IOMMU_FAULT_PAGE_REQ)
+ /* Not a recoverable page fault */
+ return -EOPNOTSUPP;
+
+ /*
+ * As long as we're holding param->lock, the queue can't be unlinked
+ * from the device and therefore cannot disappear.
+ */
+ iopf_param = param->iopf_param;
+ if (!iopf_param)
+ return -ENODEV;
+
+ if (!(fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE)) {
+ iopf = kzalloc(sizeof(*iopf), GFP_KERNEL);
+ if (!iopf)
+ return -ENOMEM;
+
+ iopf->fault = *fault;
+
+ /* Non-last request of a group. Postpone until the last one */
+ list_add(&iopf->list, &iopf_param->partial);
+
+ return 0;
+ }
+
+ group = kzalloc(sizeof(*group), GFP_KERNEL);
+ if (!group) {
+ /*
+ * The caller will send a response to the hardware. But we do
+ * need to clean up before leaving, otherwise partial faults
+ * will be stuck.
+ */
+ ret = -ENOMEM;
+ goto cleanup_partial;
+ }
+
+ group->dev = dev;
+ group->last_fault.fault = *fault;
+ INIT_LIST_HEAD(&group->faults);
+ list_add(&group->last_fault.list, &group->faults);
+ INIT_WORK(&group->work, iopf_handle_group);
+
+ /* See if we have partial faults for this group */
+ list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) {
+ if (iopf->fault.prm.grpid == fault->prm.grpid)
+ /* Insert *before* the last fault */
+ list_move(&iopf->list, &group->faults);
+ }
+
+ queue_work(iopf_param->queue->wq, &group->work);
+ return 0;
+
+cleanup_partial:
+ list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) {
+ if (iopf->fault.prm.grpid == fault->prm.grpid) {
+ list_del(&iopf->list);
+ kfree(iopf);
+ }
+ }
+ return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_queue_iopf);
+
+/**
+ * iopf_queue_flush_dev - Ensure that all queued faults have been processed
+ * @dev: the endpoint whose faults need to be flushed.
+ *
+ * The IOMMU driver calls this before releasing a PASID, to ensure that all
+ * pending faults for this PASID have been handled, and won't hit the address
+ * space of the next process that uses this PASID. The driver must make sure
+ * that no new fault is added to the queue. In particular it must flush its
+ * low-level queue before calling this function.
+ *
+ * Return: 0 on success and <0 on error.
+ */
+int iopf_queue_flush_dev(struct device *dev)
+{
+ int ret = 0;
+ struct iopf_device_param *iopf_param;
+ struct dev_iommu *param = dev->iommu;
+
+ if (!param)
+ return -ENODEV;
+
+ mutex_lock(¶m->lock);
+ iopf_param = param->iopf_param;
+ if (iopf_param)
+ flush_workqueue(iopf_param->queue->wq);
+ else
+ ret = -ENODEV;
+ mutex_unlock(¶m->lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(iopf_queue_flush_dev);
+
+/**
+ * iopf_queue_discard_partial - Remove all pending partial fault
+ * @queue: the queue whose partial faults need to be discarded
+ *
+ * When the hardware queue overflows, last page faults in a group may have been
+ * lost and the IOMMU driver calls this to discard all partial faults. The
+ * driver shouldn't be adding new faults to this queue concurrently.
+ *
+ * Return: 0 on success and <0 on error.
+ */
+int iopf_queue_discard_partial(struct iopf_queue *queue)
+{
+ struct iopf_fault *iopf, *next;
+ struct iopf_device_param *iopf_param;
+
+ if (!queue)
+ return -EINVAL;
+
+ mutex_lock(&queue->lock);
+ list_for_each_entry(iopf_param, &queue->devices, queue_list) {
+ list_for_each_entry_safe(iopf, next, &iopf_param->partial,
+ list) {
+ list_del(&iopf->list);
+ kfree(iopf);
+ }
+ }
+ mutex_unlock(&queue->lock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(iopf_queue_discard_partial);
+
+/**
+ * iopf_queue_add_device - Add producer to the fault queue
+ * @queue: IOPF queue
+ * @dev: device to add
+ *
+ * Return: 0 on success and <0 on error.
+ */
+int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev)
+{
+ int ret = -EBUSY;
+ struct iopf_device_param *iopf_param;
+ struct dev_iommu *param = dev->iommu;
+
+ if (!param)
+ return -ENODEV;
+
+ iopf_param = kzalloc(sizeof(*iopf_param), GFP_KERNEL);
+ if (!iopf_param)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&iopf_param->partial);
+ iopf_param->queue = queue; iopf_param->dev = dev;
+
+ mutex_lock(&queue->lock);
+ mutex_lock(¶m->lock);
+ if (!param->iopf_param) {
+ list_add(&iopf_param->queue_list, &queue->devices);
+ param->iopf_param = iopf_param;
+ ret = 0;
+ }
+ mutex_unlock(¶m->lock);
+ mutex_unlock(&queue->lock);
+
+ if (ret)
+ kfree(iopf_param);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(iopf_queue_add_device);
+
+/**
+ * iopf_queue_remove_device - Remove producer from fault queue
+ * @queue: IOPF queue
+ * @dev: device to remove
+ *
+ * Caller makes sure that no more faults are reported for this device.
+ *
+ * Return: 0 on success and <0 on error.
+ */
+int iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev)
+{
+ int ret = 0;
+ struct iopf_fault *iopf, *next;
+ struct iopf_device_param *iopf_param;
+ struct dev_iommu *param = dev->iommu;
+
+ if (!param || !queue)
+ return -EINVAL;
+
+ mutex_lock(&queue->lock);
+ mutex_lock(¶m->lock);
+ iopf_param = param->iopf_param;
+ if (iopf_param && iopf_param->queue == queue) {
+ list_del(&iopf_param->queue_list);
+ param->iopf_param = NULL;
+ } else {
+ ret = -EINVAL;
+ }
+ mutex_unlock(¶m->lock);
+ mutex_unlock(&queue->lock);
+ if (ret)
+ return ret;
+
+ /* Just in case some faults are still stuck */
+ list_for_each_entry_safe(iopf, next, &iopf_param->partial, list)
+ kfree(iopf);
+
+ kfree(iopf_param);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(iopf_queue_remove_device);
+
+/**
+ * iopf_queue_alloc - Allocate and initialize a fault queue
+ * @name: a unique string identifying the queue (for workqueue)
+ *
+ * Return: the queue on success and NULL on error.
+ */
+struct iopf_queue *iopf_queue_alloc(const char *name)
+{
+ struct iopf_queue *queue;
+
+ queue = kzalloc(sizeof(*queue), GFP_KERNEL);
+ if (!queue)
+ return NULL;
+
+ /*
+ * The WQ is unordered because the low-level handler enqueues faults by
+ * group. PRI requests within a group have to be ordered, but once
+ * that's dealt with, the high-level function can handle groups out of
+ * order.
+ */
+ queue->wq = alloc_workqueue("iopf_queue/%s", WQ_UNBOUND, 0, name);
+ if (!queue->wq) {
+ kfree(queue);
+ return NULL;
+ }
+
+ INIT_LIST_HEAD(&queue->devices);
+ mutex_init(&queue->lock);
+
+ return queue;
+}
+EXPORT_SYMBOL_GPL(iopf_queue_alloc);
+
+/**
+ * iopf_queue_free - Free IOPF queue
+ * @queue: queue to free
+ *
+ * Counterpart to iopf_queue_alloc(). The driver must not be queuing faults or
+ * adding/removing devices on this queue anymore.
+ */
+void iopf_queue_free(struct iopf_queue *queue)
+{
+ struct iopf_device_param *iopf_param, *next;
+
+ if (!queue)
+ return;
+
+ list_for_each_entry_safe(iopf_param, next, &queue->devices, queue_list)
+ iopf_queue_remove_device(queue, iopf_param->dev);
+
+ destroy_workqueue(queue->wq);
+ kfree(queue);
+}
+EXPORT_SYMBOL_GPL(iopf_queue_free);
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 05/24] arm64: mm: Add asid_gen_match() helper
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Add a macro to check if an ASID is from the current generation, since a
subsequent patch will introduce a third user for this test.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
arch/arm64/mm/context.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 9b26f9a88724..d702d60e64da 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -92,6 +92,9 @@ static void set_reserved_asid_bits(void)
bitmap_clear(asid_map, 0, NUM_USER_ASIDS);
}
+#define asid_gen_match(asid) \
+ (!(((asid) ^ atomic64_read(&asid_generation)) >> asid_bits))
+
static void flush_context(void)
{
int i;
@@ -220,8 +223,7 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu)
* because atomic RmWs are totally ordered for a given location.
*/
old_active_asid = atomic64_read(&per_cpu(active_asids, cpu));
- if (old_active_asid &&
- !((asid ^ atomic64_read(&asid_generation)) >> asid_bits) &&
+ if (old_active_asid && asid_gen_match(asid) &&
atomic64_cmpxchg_relaxed(&per_cpu(active_asids, cpu),
old_active_asid, asid))
goto switch_mm_fastpath;
@@ -229,7 +231,7 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu)
raw_spin_lock_irqsave(&cpu_asid_lock, flags);
/* Check that our ASID belongs to the current generation. */
asid = atomic64_read(&mm->context.id);
- if ((asid ^ atomic64_read(&asid_generation)) >> asid_bits) {
+ if (!asid_gen_match(asid)) {
asid = new_context(mm);
atomic64_set(&mm->context.id, asid);
}
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 03/24] iommu/sva: Add PASID helpers
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Let IOMMU drivers allocate a single PASID per mm. Store the mm in the
IOASID set to allow refcounting and searching mm by PASID, when handling
an I/O page fault.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/Kconfig | 5 +++
drivers/iommu/Makefile | 1 +
drivers/iommu/iommu-sva.h | 15 +++++++
drivers/iommu/iommu-sva.c | 85 +++++++++++++++++++++++++++++++++++++++
4 files changed, 106 insertions(+)
create mode 100644 drivers/iommu/iommu-sva.h
create mode 100644 drivers/iommu/iommu-sva.c
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 2ab07ce17abb..d9fa5b410015 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -102,6 +102,11 @@ config IOMMU_DMA
select IRQ_MSI_IOMMU
select NEED_SG_DMA_LENGTH
+# Shared Virtual Addressing library
+config IOMMU_SVA
+ bool
+ select IOASID
+
config FSL_PAMU
bool "Freescale IOMMU support"
depends on PCI
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 9f33fdb3bb05..40c800dd4e3e 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -37,3 +37,4 @@ obj-$(CONFIG_S390_IOMMU) += s390-iommu.o
obj-$(CONFIG_QCOM_IOMMU) += qcom_iommu.o
obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o
obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o
+obj-$(CONFIG_IOMMU_SVA) += iommu-sva.o
diff --git a/drivers/iommu/iommu-sva.h b/drivers/iommu/iommu-sva.h
new file mode 100644
index 000000000000..78f806fcacbe
--- /dev/null
+++ b/drivers/iommu/iommu-sva.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * SVA library for IOMMU drivers
+ */
+#ifndef _IOMMU_SVA_H
+#define _IOMMU_SVA_H
+
+#include <linux/ioasid.h>
+#include <linux/mm_types.h>
+
+int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max);
+void iommu_sva_free_pasid(struct mm_struct *mm);
+struct mm_struct *iommu_sva_find(ioasid_t pasid);
+
+#endif /* _IOMMU_SVA_H */
diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
new file mode 100644
index 000000000000..442644a1ade0
--- /dev/null
+++ b/drivers/iommu/iommu-sva.c
@@ -0,0 +1,85 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Helpers for IOMMU drivers implementing SVA
+ */
+#include <linux/mutex.h>
+#include <linux/sched/mm.h>
+
+#include "iommu-sva.h"
+
+static DEFINE_MUTEX(iommu_sva_lock);
+static DECLARE_IOASID_SET(shared_pasid);
+
+/**
+ * iommu_sva_alloc_pasid - Allocate a PASID for the mm
+ * @mm: the mm
+ * @min: minimum PASID value (inclusive)
+ * @max: maximum PASID value (inclusive)
+ *
+ * Try to allocate a PASID for this mm, or take a reference to the existing one
+ * provided it fits within the [min, max] range. On success the PASID is
+ * available in mm->pasid, and must be released with iommu_sva_free_pasid().
+ *
+ * Returns 0 on success and < 0 on error.
+ */
+int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max)
+{
+ int ret = 0;
+ ioasid_t pasid;
+
+ if (min == INVALID_IOASID || max == INVALID_IOASID ||
+ min == 0 || max < min)
+ return -EINVAL;
+
+ mutex_lock(&iommu_sva_lock);
+ if (mm->pasid) {
+ if (mm->pasid >= min && mm->pasid <= max)
+ ioasid_get(mm->pasid);
+ else
+ ret = -EOVERFLOW;
+ } else {
+ pasid = ioasid_alloc(&shared_pasid, min, max, mm);
+ if (pasid == INVALID_IOASID)
+ ret = -ENOMEM;
+ else
+ mm->pasid = pasid;
+ }
+ mutex_unlock(&iommu_sva_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_sva_alloc_pasid);
+
+/**
+ * iommu_sva_free_pasid - Release the mm's PASID
+ * @mm: the mm.
+ *
+ * Drop one reference to a PASID allocated with iommu_sva_alloc_pasid()
+ */
+void iommu_sva_free_pasid(struct mm_struct *mm)
+{
+ mutex_lock(&iommu_sva_lock);
+ if (ioasid_put(mm->pasid))
+ mm->pasid = 0;
+ mutex_unlock(&iommu_sva_lock);
+}
+EXPORT_SYMBOL_GPL(iommu_sva_free_pasid);
+
+/* ioasid wants a void * argument */
+static bool __mmget_not_zero(void *mm)
+{
+ return mmget_not_zero(mm);
+}
+
+/**
+ * iommu_sva_find() - Find mm associated to the given PASID
+ * @pasid: Process Address Space ID assigned to the mm
+ *
+ * On success a reference to the mm is taken, and must be released with mmput().
+ *
+ * Returns the mm corresponding to this PASID, or an error if not found.
+ */
+struct mm_struct *iommu_sva_find(ioasid_t pasid)
+{
+ return ioasid_find(&shared_pasid, pasid, __mmget_not_zero);
+}
+EXPORT_SYMBOL_GPL(iommu_sva_find);
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 02/24] iommu/ioasid: Add ioasid references
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Let IOASID users take references to existing ioasids with ioasid_get().
ioasid_put() drops a reference and only frees the ioasid when its
reference number is zero. It returns true if the ioasid was freed.
For drivers that don't call ioasid_get(), ioasid_put() is the same as
ioasid_free().
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
v6->v7: rename ioasid_free() to ioasid_put(), add WARN in ioasid_get()
---
include/linux/ioasid.h | 10 ++++++++--
drivers/iommu/intel-iommu.c | 4 ++--
drivers/iommu/intel-svm.c | 6 +++---
drivers/iommu/ioasid.c | 38 +++++++++++++++++++++++++++++++++----
4 files changed, 47 insertions(+), 11 deletions(-)
diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h
index 6f000d7a0ddc..e9dacd4b9f6b 100644
--- a/include/linux/ioasid.h
+++ b/include/linux/ioasid.h
@@ -34,7 +34,8 @@ struct ioasid_allocator_ops {
#if IS_ENABLED(CONFIG_IOASID)
ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
void *private);
-void ioasid_free(ioasid_t ioasid);
+void ioasid_get(ioasid_t ioasid);
+bool ioasid_put(ioasid_t ioasid);
void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
bool (*getter)(void *));
int ioasid_register_allocator(struct ioasid_allocator_ops *allocator);
@@ -48,10 +49,15 @@ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min,
return INVALID_IOASID;
}
-static inline void ioasid_free(ioasid_t ioasid)
+static inline void ioasid_get(ioasid_t ioasid)
{
}
+static inline bool ioasid_put(ioasid_t ioasid)
+{
+ return false;
+}
+
static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid,
bool (*getter)(void *))
{
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index ed21ce6d1238..0230f35480ee 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -5432,7 +5432,7 @@ static void auxiliary_unlink_device(struct dmar_domain *domain,
domain->auxd_refcnt--;
if (!domain->auxd_refcnt && domain->default_pasid > 0)
- ioasid_free(domain->default_pasid);
+ ioasid_put(domain->default_pasid);
}
static int aux_domain_add_dev(struct dmar_domain *domain,
@@ -5494,7 +5494,7 @@ static int aux_domain_add_dev(struct dmar_domain *domain,
spin_unlock(&iommu->lock);
spin_unlock_irqrestore(&device_domain_lock, flags);
if (!domain->auxd_refcnt && domain->default_pasid > 0)
- ioasid_free(domain->default_pasid);
+ ioasid_put(domain->default_pasid);
return ret;
}
diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c
index 2998418f0a38..86f1264bd07c 100644
--- a/drivers/iommu/intel-svm.c
+++ b/drivers/iommu/intel-svm.c
@@ -353,7 +353,7 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_
if (mm) {
ret = mmu_notifier_register(&svm->notifier, mm);
if (ret) {
- ioasid_free(svm->pasid);
+ ioasid_put(svm->pasid);
kfree(svm);
kfree(sdev);
goto out;
@@ -371,7 +371,7 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_
if (ret) {
if (mm)
mmu_notifier_unregister(&svm->notifier, mm);
- ioasid_free(svm->pasid);
+ ioasid_put(svm->pasid);
kfree(svm);
kfree(sdev);
goto out;
@@ -447,7 +447,7 @@ int intel_svm_unbind_mm(struct device *dev, int pasid)
kfree_rcu(sdev, rcu);
if (list_empty(&svm->devs)) {
- ioasid_free(svm->pasid);
+ ioasid_put(svm->pasid);
if (svm->mm)
mmu_notifier_unregister(&svm->notifier, svm->mm);
list_del(&svm->list);
diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c
index 0f8dd377aada..50ee27bbd04e 100644
--- a/drivers/iommu/ioasid.c
+++ b/drivers/iommu/ioasid.c
@@ -2,7 +2,7 @@
/*
* I/O Address Space ID allocator. There is one global IOASID space, split into
* subsets. Users create a subset with DECLARE_IOASID_SET, then allocate and
- * free IOASIDs with ioasid_alloc and ioasid_free.
+ * free IOASIDs with ioasid_alloc and ioasid_put.
*/
#include <linux/ioasid.h>
#include <linux/module.h>
@@ -15,6 +15,7 @@ struct ioasid_data {
struct ioasid_set *set;
void *private;
struct rcu_head rcu;
+ refcount_t refs;
};
/*
@@ -314,6 +315,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
data->set = set;
data->private = private;
+ refcount_set(&data->refs, 1);
/*
* Custom allocator needs allocator data to perform platform specific
@@ -346,11 +348,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,
EXPORT_SYMBOL_GPL(ioasid_alloc);
/**
- * ioasid_free - Free an IOASID
+ * ioasid_get - obtain a reference to the IOASID
+ */
+void ioasid_get(ioasid_t ioasid)
+{
+ struct ioasid_data *ioasid_data;
+
+ spin_lock(&ioasid_allocator_lock);
+ ioasid_data = xa_load(&active_allocator->xa, ioasid);
+ if (ioasid_data)
+ refcount_inc(&ioasid_data->refs);
+ else
+ WARN_ON(1);
+ spin_unlock(&ioasid_allocator_lock);
+}
+EXPORT_SYMBOL_GPL(ioasid_get);
+
+/**
+ * ioasid_put - Release a reference to an ioasid
* @ioasid: the ID to remove
+ *
+ * Put a reference to the IOASID, free it when the number of references drops to
+ * zero.
+ *
+ * Return: %true if the IOASID was freed, %false otherwise.
*/
-void ioasid_free(ioasid_t ioasid)
+bool ioasid_put(ioasid_t ioasid)
{
+ bool free = false;
struct ioasid_data *ioasid_data;
spin_lock(&ioasid_allocator_lock);
@@ -360,6 +385,10 @@ void ioasid_free(ioasid_t ioasid)
goto exit_unlock;
}
+ free = refcount_dec_and_test(&ioasid_data->refs);
+ if (!free)
+ goto exit_unlock;
+
active_allocator->ops->free(ioasid, active_allocator->ops->pdata);
/* Custom allocator needs additional steps to free the xa element */
if (active_allocator->flags & IOASID_ALLOCATOR_CUSTOM) {
@@ -369,8 +398,9 @@ void ioasid_free(ioasid_t ioasid)
exit_unlock:
spin_unlock(&ioasid_allocator_lock);
+ return free;
}
-EXPORT_SYMBOL_GPL(ioasid_free);
+EXPORT_SYMBOL_GPL(ioasid_put);
/**
* ioasid_find - Find IOASID data
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 01/24] mm: Add a PASID field to mm_struct
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org>
Some devices can tag their DMA requests with a 20-bit Process Address
Space ID (PASID), allowing them to access multiple address spaces. In
combination with recoverable I/O page faults (for example PCIe PRI),
PASID allows the IOMMU to share page tables with the MMU.
To make sure that a single PASID is allocated for each address space, as
required by Intel ENQCMD, store the PASID in the mm_struct. The IOMMU
driver is in charge of serializing modifications to the PASID field.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
include/linux/mm_types.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 4aba6c0c2ba8..8db647275817 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -534,6 +534,10 @@ struct mm_struct {
atomic_long_t hugetlb_usage;
#endif
struct work_struct async_put_work;
+#ifdef CONFIG_IOMMU_SUPPORT
+ /* Address space ID used by device DMA */
+ unsigned int pasid;
+#endif
} __randomize_layout;
/*
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [PATCH v7 00/24] iommu: Shared Virtual Addressing for SMMUv3
From: Jean-Philippe Brucker @ 2020-05-19 17:54 UTC (permalink / raw)
To: iommu, devicetree, linux-arm-kernel, linux-pci, linux-mm
Cc: fenghua.yu, kevin.tian, jacob.jun.pan, jgg, catalin.marinas, joro,
robin.murphy, hch, zhangfei.gao, Jean-Philippe Brucker,
Jonathan.Cameron, felix.kuehling, xuzaibo, will, christian.koenig,
baolu.lu
Shared Virtual Addressing (SVA) allows to share process page tables with
devices using the IOMMU, PASIDs and I/O page faults. Add SVA support to
the Arm SMMUv3 driver.
Since v6 [1]:
* Rename ioasid_free() to ioasid_put() in patch 02, requiring changes to
the Intel drivers.
* Use mmu_notifier_register() in patch 16 to avoid copying the ops and
simplify the invalidate() notifier in patch 17.
* As a result, replace context spinlock with a mutex. Simplified locking in
patch 11 (That patch still looks awful, but I think the series is more
readable overall). And I've finally been able to remove the GFP_ATOMIC
allocations.
* Use a single patch (04) for io-pgfault.c, since the code was simplified
in v6. Fixed partial list in patch 04.
[1] https://lore.kernel.org/linux-iommu/20200430143424.2787566-1-jean-philippe@linaro.org/
Jean-Philippe Brucker (24):
mm: Add a PASID field to mm_struct
iommu/ioasid: Add ioasid references
iommu/sva: Add PASID helpers
iommu: Add a page fault handler
arm64: mm: Add asid_gen_match() helper
arm64: mm: Pin down ASIDs for sharing mm with devices
iommu/io-pgtable-arm: Move some definitions to a header
iommu/arm-smmu-v3: Manage ASIDs with xarray
arm64: cpufeature: Export symbol read_sanitised_ftr_reg()
iommu/arm-smmu-v3: Share process page tables
iommu/arm-smmu-v3: Seize private ASID
iommu/arm-smmu-v3: Add support for VHE
iommu/arm-smmu-v3: Enable broadcast TLB maintenance
iommu/arm-smmu-v3: Add SVA feature checking
iommu/arm-smmu-v3: Add SVA device feature
iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind()
iommu/arm-smmu-v3: Hook up ATC invalidation to mm ops
iommu/arm-smmu-v3: Add support for Hardware Translation Table Update
iommu/arm-smmu-v3: Maintain a SID->device structure
dt-bindings: document stall property for IOMMU masters
iommu/arm-smmu-v3: Add stall support for platform devices
PCI/ATS: Add PRI stubs
PCI/ATS: Export PRI functions
iommu/arm-smmu-v3: Add support for PRI
drivers/iommu/Kconfig | 12 +
drivers/iommu/Makefile | 2 +
.../devicetree/bindings/iommu/iommu.txt | 18 +
arch/arm64/include/asm/mmu.h | 1 +
arch/arm64/include/asm/mmu_context.h | 11 +-
drivers/iommu/io-pgtable-arm.h | 30 +
drivers/iommu/iommu-sva.h | 15 +
include/linux/ioasid.h | 10 +-
include/linux/iommu.h | 53 +
include/linux/mm_types.h | 4 +
include/linux/pci-ats.h | 8 +
arch/arm64/kernel/cpufeature.c | 1 +
arch/arm64/mm/context.c | 103 +-
drivers/iommu/arm-smmu-v3.c | 1552 +++++++++++++++--
drivers/iommu/intel-iommu.c | 4 +-
drivers/iommu/intel-svm.c | 6 +-
drivers/iommu/io-pgfault.c | 459 +++++
drivers/iommu/io-pgtable-arm.c | 27 +-
drivers/iommu/ioasid.c | 38 +-
drivers/iommu/iommu-sva.c | 85 +
drivers/iommu/of_iommu.c | 5 +-
drivers/pci/ats.c | 4 +
MAINTAINERS | 3 +-
23 files changed, 2286 insertions(+), 165 deletions(-)
create mode 100644 drivers/iommu/io-pgtable-arm.h
create mode 100644 drivers/iommu/iommu-sva.h
create mode 100644 drivers/iommu/io-pgfault.c
create mode 100644 drivers/iommu/iommu-sva.c
--
2.26.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH 2/2] ARM: Allow either FLATMEM or SPARSEMEM on the multiplatform build
From: Mike Rapoport @ 2020-05-19 17:59 UTC (permalink / raw)
To: Russell King - ARM Linux admin
Cc: Doug Berger, Arnd Bergmann, Stephen Boyd, Kevin Cernekee,
Florian Fainelli, Gregory Fong, Linux ARM
In-Reply-To: <20200519165445.GI1551@shell.armlinux.org.uk>
Hi Russell,
On Tue, May 19, 2020 at 05:54:45PM +0100, Russell King - ARM Linux admin wrote:
> On Tue, May 19, 2020 at 05:32:52PM +0200, Arnd Bergmann wrote:
> > On Tue, May 19, 2020 at 5:27 PM Mike Rapoport <rppt@linux.ibm.com> wrote:
> > > On Tue, May 19, 2020 at 04:04:52PM +0100, Russell King - ARM Linux admin wrote:
> > > > There are cases where the reason to use sparsemem is not an optional
> > > > one but is out of necessity - they require the page array to be split
> > > > up in order to boot successfully.
> > > >
> > > > With that in mind, flatmem becomes an "optimisation" over sparsemem.
> > >
> > > At the moment, there are three platforms that enable SPARSEMEM: ARCH_EP93XX,
> > > ARCH_RPC and ARCH_SA1100. All the rest have FLATMEM implcitly selected.
> > >
> > > I do not intend to change that, I am only going add an ability to select
> > > the memory model for ARCH_MULTIPLATFORM.
> > >
> > > I'll respin the series on the list before adding the patches to the
> > > patch system.
> >
> > I think we'll make EP93xx part of multiplatform at some point, IIRC
> > only the missing clock driver is stopping us at the moment, and I already
> > discussed with Linus Walleij how that can be done.
> >
> > My guess is that ep93xx is one platform on which sparsemem is
> > just an optimization to reduce the initial memory consumption, but
> > we should verify that when we get there.
>
> When you have a platform where the memory is segmented into separate
> blocks widely spaced, sparsemem is not an optimisation. For example,
> If you have four blocks spaced across 1GB, that requires about
> 256Ki struct page's. Assuming 32 byte struct page, that requires 8MiB
> of contiguous memory.
>
> If we also consider the other constraint - that the kernel has to fit
> in the first bank of memory, then, considering the size of the kernel
> (the first two are non-multiplatform kernels):
>
> text data bss dec hex filename
> 4045505 903700 92984 5042189 4cf00d n2100/boot/vmlinux-5.6.12+
> 4045361 957276 1159052 6161689 5e0519 assabet/boot/vmlinux-5.2.0+
> 6933363 1451420 203984 8588767 830ddf virt/boot/vmlinux-5.6.0+
> 9980260 3568070 7403296 20951626 13fb24a multi/boot/vmlinux-5.3.0+
>
> So, realistically, we're looking at imposing a requirement that the
> first bank of memory is no smaller than 16MB on these machines if a
> "default" flatmem multiplatform kernel is going to be able to boot,
> and if all banks are populated, that there is another bank that has
> at least 8MB to hold the memmap array.
>
> BTW, the "optimisation" argument for sparsemem doesn't actually
> hold. For flatmem, we free the unused parts of the memmap array,
> freeing those pages for other uses. Sparsemem on ARM is about
> allowing these platforms to boot with their memory spread across
> the physical address space savannah.
There is no argument that sparsemem is more robust and flexible in its
ability to hande separated memory banks. And, AFAIU, nobody suggests to
start using flatmem instead of sparsemem on any machine.
Currently, when you build the kernel with "make defconfig && make" or
"make multi_X_defconfig && make" kconfig implicitly selects flatmem and
there is no possiblity to enable sparsemem in the menuconfig.
Florian sent the pathces with the purpose to allow overriding the
explict choise of flatmem with sparsemem in menuconfig.
At some point I've asked whether we would like to use sparsemem instead
of flatmem as the default for the multiplafrom build. The upside of such
change is the robustness you described above. The downside is a bit
larger kernel image and slight slow down in page_to_pfn and pfn_to_page.
That said we have two options:
* Massage the patches Florian sent and add ability to select memory
model in menuconfig for the multiplatform builds without changing
anything in configurations that were not manually atered.
* Simply add 'select ARCH_SPAPSEMEM_ENABLE' under 'ARCH_MULTIPLAFROM'
and force all multiplatform builds to use sparsemem.
So the question is what is the preferred way to move forward.
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
--
Sincerely yours,
Mike.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH v2 1/3] mm/memory-failure: Add memory_failure_queue_kick()
From: Rafael J. Wysocki @ 2020-05-19 17:53 UTC (permalink / raw)
To: HORIGUCHI NAOYA(堀口 直也), James Morse
Cc: Mark Rutland, Tony Luck, linux-acpi@vger.kernel.org,
Catalin Marinas, Xie XiuQi, Rafael J. Wysocki, linux-mm@kvack.org,
Tyler Baicar, Borislav Petkov, Andrew Morton, Will Deacon,
Naoya Horiguchi, linux-arm-kernel@lists.infradead.org, Len Brown
In-Reply-To: <20200519031511.GA31023@hori.linux.bs1.fc.nec.co.jp>
On Tue, May 19, 2020 at 5:15 AM HORIGUCHI NAOYA(堀口 直也)
<naoya.horiguchi@nec.com> wrote:
>
> On Mon, May 18, 2020 at 12:58:28PM -0700, Andrew Morton wrote:
> > On Mon, 18 May 2020 14:45:05 +0200 "Rafael J. Wysocki" <rjw@rjwysocki.net> wrote:
> >
> > > On Friday, May 1, 2020 6:45:41 PM CEST James Morse wrote:
> > > > The GHES code calls memory_failure_queue() from IRQ context to schedule
> > > > work on the current CPU so that memory_failure() can sleep.
> > > >
> > > > For synchronous memory errors the arch code needs to know any signals
> > > > that memory_failure() will trigger are pending before it returns to
> > > > user-space, possibly when exiting from the IRQ.
> > > >
> > > > Add a helper to kick the memory failure queue, to ensure the scheduled
> > > > work has happened. This has to be called from process context, so may
> > > > have been migrated from the original cpu. Pass the cpu the work was
> > > > queued on.
> > > >
> > > > Change memory_failure_work_func() to permit being called on the 'wrong'
> > > > cpu.
> > > >
> > > > --- a/include/linux/mm.h
> > > > +++ b/include/linux/mm.h
> > > > @@ -3012,6 +3012,7 @@ enum mf_flags {
> > > > };
> > > > extern int memory_failure(unsigned long pfn, int flags);
> > > > extern void memory_failure_queue(unsigned long pfn, int flags);
> > > > +extern void memory_failure_queue_kick(int cpu);
> > > > extern int unpoison_memory(unsigned long pfn);
> > > > extern int get_hwpoison_page(struct page *page);
> > > > #define put_hwpoison_page(page) put_page(page)
> > > > diff --git a/mm/memory-failure.c b/mm/memory-failure.c
> > > > index a96364be8ab4..c4afb407bf0f 100644
> > > > --- a/mm/memory-failure.c
> > > > +++ b/mm/memory-failure.c
> > > > @@ -1493,7 +1493,7 @@ static void memory_failure_work_func(struct work_struct *work)
> > > > unsigned long proc_flags;
> > > > int gotten;
> > > >
> > > > - mf_cpu = this_cpu_ptr(&memory_failure_cpu);
> > > > + mf_cpu = container_of(work, struct memory_failure_cpu, work);
> > > > for (;;) {
> > > > spin_lock_irqsave(&mf_cpu->lock, proc_flags);
> > > > gotten = kfifo_get(&mf_cpu->fifo, &entry);
> > > > @@ -1507,6 +1507,19 @@ static void memory_failure_work_func(struct work_struct *work)
> > > > }
> > > > }
> > > >
> > > > +/*
> > > > + * Process memory_failure work queued on the specified CPU.
> > > > + * Used to avoid return-to-userspace racing with the memory_failure workqueue.
> > > > + */
> > > > +void memory_failure_queue_kick(int cpu)
> > > > +{
> > > > + struct memory_failure_cpu *mf_cpu;
> > > > +
> > > > + mf_cpu = &per_cpu(memory_failure_cpu, cpu);
> > > > + cancel_work_sync(&mf_cpu->work);
> > > > + memory_failure_work_func(&mf_cpu->work);
> > > > +}
> > > > +
> > > > static int __init memory_failure_init(void)
> > > > {
> > > > struct memory_failure_cpu *mf_cpu;
> > > >
> > >
> > > I could apply this provided an ACK from the mm people.
> > >
> >
> > Naoya Horiguchi is the memory-failure.c person. A review would be
> > appreciated please?
> >
> > I'm struggling with it a bit. memory_failure_queue_kick() should be
> > called on the cpu which is identified by arg `cpu', yes?
> > memory_failure_work_func() appears to assume this.
> >
> > If that's right then a) why bother passing in the `cpu' arg? and b)
> > what keeps this thread pinned to that CPU? cancel_work_sync() can
> > schedule.
>
> If I read correctly, memory_failure work is queue on the CPU on which the
> user process ran when it touched the corrupted memory, and the process can
> be scheduled on another CPU when the kernel returned back to userspace after
> handling the GHES event. So we need to remember where the memory_failure
> event is queued to flush proper work queue. So I feel that this properly
> implements it.
>
> Considering the effect to the other caller, currently memory_failure_queue()
> has 2 callers, ghes_handle_memory_failure() and cec_add_elem(). The former
> is what we try to change now. And the latter is to execute soft offline
> (which is related to corrected non-fatal errors), so that's not affected by
> the reported issue. So I don't think that this change breaks the other
> caller.
>
> So I'm fine with the suggested change.
>
> Acked-by: Naoya Horiguchi <naoya.horiguchi@nec.com>
OK, thanks!
So because patch [1/3] has been ACKed already, I'm applying this
series as 5.8 material.
Thanks everyone!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH 00/11] arm/arm64: Turning IPIs into normal interrupts
From: Florian Fainelli @ 2020-05-19 17:50 UTC (permalink / raw)
To: Marc Zyngier, linux-arm-kernel, linux-kernel
Cc: Sumit Garg, Russell King, Jason Cooper, Will Deacon,
Catalin Marinas, Thomas Gleixner, kernel-team
In-Reply-To: <20200519161755.209565-1-maz@kernel.org>
On 5/19/2020 9:17 AM, Marc Zyngier wrote:
> For as long as SMP ARM has existed, IPIs have been handled as
> something special. The arch code and the interrupt controller exchange
> a couple of hooks (one to generate an IPI, another to handle it).
>
> Although this is perfectly manageable, it prevents the use of features
> that we could use if IPIs were Linux IRQs (such as pseudo-NMIs). It
> also means that each interrupt controller driver has to follow an
> architecture-specific interface instead of just implementing the base
> irqchip functionnalities. The arch code also duplicates a number of
> things that the core irq code already does (such as calling
> set_irq_regs(), irq_enter()...).
>
> This series tries to remedy this on arm/arm64 by offering a new
> registration interface where the irqchip gives the arch code a range
> of interrupts to use for IPIs. The arch code requests these as normal
> interrupts.
>
> The bulk of the work is at the interrupt controller level, where all 3
> irqchips used on arm64 get converted.
>
> Finally, the arm64 code drops the legacy registration interface. The
> same thing could be done on 32bit as well once the two remaining
> irqchips using that interface get converted.
>
> There is probably more that could be done: statistics are still
> architecture-private code, for example, and no attempt is made to
> solve that (apart from hidding the IRQs from /proc/interrupt).
>
> This has been tested on a bunch of 32 and 64bit guests.
Does this patch series change your position on this patch series
https://lore.kernel.org/linux-arm-kernel/20191023000547.7831-3-f.fainelli@gmail.com/T/
or is this still a no-no?
>
> Marc Zyngier (11):
> genirq: Add fasteoi IPI flow
> genirq: Allow interrupts to be excluded from /proc/interrupts
> arm64: Allow IPIs to be handled as normal interrupts
> ARM: Allow IPIs to be handled as normal interrupts
> irqchip/gic-v3: Describe the SGI range
> irqchip/gic-v3: Configure SGIs as standard interrupts
> irqchip/gic: Refactor SMP configuration
> irqchip/gic: Configure SGIs as standard interrupts
> irqchip/gic-common: Don't enable SGIs by default
> irqchip/bcm2836: Configure mailbox interrupts as standard interrupts
> arm64: Kill __smp_cross_call and co
>
> arch/arm/Kconfig | 1 +
> arch/arm/include/asm/smp.h | 5 +
> arch/arm/kernel/smp.c | 97 +++++++++++---
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/irq_work.h | 4 +-
> arch/arm64/include/asm/smp.h | 6 +-
> arch/arm64/kernel/smp.c | 98 +++++++++++----
> drivers/irqchip/irq-bcm2836.c | 151 ++++++++++++++++++----
> drivers/irqchip/irq-gic-common.c | 3 -
> drivers/irqchip/irq-gic-v3.c | 109 ++++++++++------
> drivers/irqchip/irq-gic.c | 203 ++++++++++++++++++------------
> include/linux/irq.h | 4 +-
> kernel/irq/chip.c | 26 ++++
> kernel/irq/debugfs.c | 1 +
> kernel/irq/proc.c | 2 +-
> kernel/irq/settings.h | 7 ++
> 16 files changed, 515 insertions(+), 203 deletions(-)
>
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH 1/1] soc: ti: omap-prm: use atomic iopoll instead of sleeping one
From: Tony Lindgren @ 2020-05-19 17:45 UTC (permalink / raw)
To: Tero Kristo; +Cc: s-anna, linux-omap, linux-kernel, linux-arm-kernel, ssantosh
In-Reply-To: <20200514073718.17690-1-t-kristo@ti.com>
* Tero Kristo <t-kristo@ti.com> [200514 00:38]:
> The reset handling APIs for omap-prm can be invoked PM runtime which
> runs in atomic context. For this to work properly, switch to atomic
> iopoll version instead of the current which can sleep. Otherwise,
> this throws a "BUG: scheduling while atomic" warning. Issue is seen
> rather easily when CONFIG_PREEMPT is enabled.
>
> Signed-off-by: Tero Kristo <t-kristo@ti.com>
Santosh do you want me to pick this for fixes?
Regards,
Tony
> ---
> drivers/soc/ti/omap_prm.c | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/soc/ti/omap_prm.c b/drivers/soc/ti/omap_prm.c
> index 96c6f777519c..c9b3f9ebf0bb 100644
> --- a/drivers/soc/ti/omap_prm.c
> +++ b/drivers/soc/ti/omap_prm.c
> @@ -256,10 +256,10 @@ static int omap_reset_deassert(struct reset_controller_dev *rcdev,
> goto exit;
>
> /* wait for the status to be set */
> - ret = readl_relaxed_poll_timeout(reset->prm->base +
> - reset->prm->data->rstst,
> - v, v & BIT(st_bit), 1,
> - OMAP_RESET_MAX_WAIT);
> + ret = readl_relaxed_poll_timeout_atomic(reset->prm->base +
> + reset->prm->data->rstst,
> + v, v & BIT(st_bit), 1,
> + OMAP_RESET_MAX_WAIT);
> if (ret)
> pr_err("%s: timedout waiting for %s:%lu\n", __func__,
> reset->prm->data->name, id);
> --
> 2.17.1
>
> --
> Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH v9] arm64: dts: qcom: sc7180: Add WCN3990 WLAN module device node
From: Evan Green @ 2020-05-19 17:28 UTC (permalink / raw)
To: Rakesh Pillai
Cc: open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS, LKML,
linux-arm Mailing List, linux-arm-msm
In-Reply-To: <1589712411-26718-1-git-send-email-pillair@codeaurora.org>
On Sun, May 17, 2020 at 3:47 AM Rakesh Pillai <pillair@codeaurora.org> wrote:
>
> Add device node for the ath10k SNOC platform driver probe
> and add resources required for WCN3990 on sc7180 soc.
>
> Signed-off-by: Rakesh Pillai <pillair@codeaurora.org>
> ---
> Changes from v8:
> - Removed the qcom,msa-fixed-perm
> ---
> arch/arm64/boot/dts/qcom/sc7180-idp.dts | 7 +++++++
> arch/arm64/boot/dts/qcom/sc7180.dtsi | 27 +++++++++++++++++++++++++++
> 2 files changed, 34 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/qcom/sc7180-idp.dts b/arch/arm64/boot/dts/qcom/sc7180-idp.dts
> index 4e9149d..38b102e 100644
> --- a/arch/arm64/boot/dts/qcom/sc7180-idp.dts
> +++ b/arch/arm64/boot/dts/qcom/sc7180-idp.dts
> @@ -389,6 +389,13 @@
> };
> };
>
> +&wifi {
> + status = "okay";
> + wifi-firmware {
> + iommus = <&apps_smmu 0xc2 0x1>;
> + };
> +};
> +
> /* PINCTRL - additions to nodes defined in sc7180.dtsi */
>
> &qspi_clk {
> diff --git a/arch/arm64/boot/dts/qcom/sc7180.dtsi b/arch/arm64/boot/dts/qcom/sc7180.dtsi
> index f1280e0..dd4e095 100644
> --- a/arch/arm64/boot/dts/qcom/sc7180.dtsi
> +++ b/arch/arm64/boot/dts/qcom/sc7180.dtsi
> @@ -63,6 +63,11 @@
> clock-frequency = <32764>;
> #clock-cells = <0>;
> };
> +
> + wlan_fw_mem: memory@94100000 {
> + reg = <0 0x94100000 0 0x200000>;
> + no-map;
> + };
This node is not in the right place. Up through v8, this lived inside
reserved-memory. Here it seems to apply into the clocks {} node, which
is not the right spot.
> };
>
> reserved_memory: reserved-memory {
> @@ -944,6 +949,28 @@
> };
> };
>
> + wifi: wifi@18800000 {
> + compatible = "qcom,wcn3990-wifi";
> + reg = <0 0x18800000 0 0x800000>;
> + reg-names = "membase";
> + iommus = <&apps_smmu 0xc0 0x1>;
> + interrupts =
> + <GIC_SPI 414 IRQ_TYPE_LEVEL_HIGH /* CE0 */ >,
> + <GIC_SPI 415 IRQ_TYPE_LEVEL_HIGH /* CE1 */ >,
> + <GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH /* CE2 */ >,
> + <GIC_SPI 417 IRQ_TYPE_LEVEL_HIGH /* CE3 */ >,
> + <GIC_SPI 418 IRQ_TYPE_LEVEL_HIGH /* CE4 */ >,
> + <GIC_SPI 419 IRQ_TYPE_LEVEL_HIGH /* CE5 */ >,
> + <GIC_SPI 420 IRQ_TYPE_LEVEL_HIGH /* CE6 */ >,
> + <GIC_SPI 421 IRQ_TYPE_LEVEL_HIGH /* CE7 */ >,
> + <GIC_SPI 422 IRQ_TYPE_LEVEL_HIGH /* CE8 */ >,
> + <GIC_SPI 423 IRQ_TYPE_LEVEL_HIGH /* CE9 */ >,
> + <GIC_SPI 424 IRQ_TYPE_LEVEL_HIGH /* CE10 */>,
> + <GIC_SPI 425 IRQ_TYPE_LEVEL_HIGH /* CE11 */>;
> + memory-region = <&wlan_fw_mem>;
Should any of the *-supply regulators be specified? Or are they all
board specific? Or just not needed? The example has these:
vdd-0.8-cx-mx-supply = <&pm8998_l5>;
vdd-1.8-xo-supply = <&vreg_l7a_1p8>;
vdd-1.3-rfa-supply = <&vreg_l17a_1p3>;
vdd-3.3-ch0-supply = <&vreg_l25a_3p3>;
> + status = "disabled";
> + };
> +
> config_noc: interconnect@1500000 {
> compatible = "qcom,sc7180-config-noc";
> reg = <0 0x01500000 0 0x28000>;
> --
> 2.7.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH v3 4/4] PCI: cadence: Use "dma-ranges" instead of "cdns,no-bar-match-nbits" property
From: Rob Herring @ 2020-05-19 17:11 UTC (permalink / raw)
To: Kishon Vijay Abraham I
Cc: devicetree, Lorenzo Pieralisi, PCI, linux-kernel@vger.kernel.org,
Tom Joseph, Bjorn Helgaas, Robin Murphy,
moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE
In-Reply-To: <20200508130646.23939-5-kishon@ti.com>
On Fri, May 8, 2020 at 7:07 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Cadence PCIe core driver (host mode) uses "cdns,no-bar-match-nbits"
> property to configure the number of bits passed through from PCIe
> address to internal address in Inbound Address Translation register.
> This only used the NO MATCH BAR.
>
> However standard PCI dt-binding already defines "dma-ranges" to
> describe the address ranges accessible by PCIe controller. Add support
> in Cadence PCIe host driver to parse dma-ranges and configure the
> inbound regions for BAR0, BAR1 and NO MATCH BAR. Cadence IP specifies
> maximum size for BAR0 as 256GB, maximum size for BAR1 as 2 GB, so if
> the dma-ranges specifies a size larger than the maximum allowed, the
> driver will split and configure the BARs.
Would be useful to know what your dma-ranges contains now.
> Legacy device tree binding compatibility is maintained by retaining
> support for "cdns,no-bar-match-nbits".
>
> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
> ---
> .../controller/cadence/pcie-cadence-host.c | 141 ++++++++++++++++--
> drivers/pci/controller/cadence/pcie-cadence.h | 17 ++-
> 2 files changed, 141 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/pci/controller/cadence/pcie-cadence-host.c b/drivers/pci/controller/cadence/pcie-cadence-host.c
> index 6ecebb79057a..2485ecd8434d 100644
> --- a/drivers/pci/controller/cadence/pcie-cadence-host.c
> +++ b/drivers/pci/controller/cadence/pcie-cadence-host.c
> @@ -11,6 +11,12 @@
>
> #include "pcie-cadence.h"
>
> +static u64 cdns_rp_bar_max_size[] = {
> + [RP_BAR0] = _ULL(128 * SZ_2G),
> + [RP_BAR1] = SZ_2G,
> + [RP_NO_BAR] = SZ_64T,
> +};
> +
> void __iomem *cdns_pci_map_bus(struct pci_bus *bus, unsigned int devfn,
> int where)
> {
> @@ -106,6 +112,117 @@ static int cdns_pcie_host_init_root_port(struct cdns_pcie_rc *rc)
> return 0;
> }
>
> +static void cdns_pcie_host_bar_ib_config(struct cdns_pcie_rc *rc,
> + enum cdns_pcie_rp_bar bar,
> + u64 cpu_addr, u32 aperture)
> +{
> + struct cdns_pcie *pcie = &rc->pcie;
> + u32 addr0, addr1;
> +
> + addr0 = CDNS_PCIE_AT_IB_RP_BAR_ADDR0_NBITS(aperture) |
> + (lower_32_bits(cpu_addr) & GENMASK(31, 8));
> + addr1 = upper_32_bits(cpu_addr);
> + cdns_pcie_writel(pcie, CDNS_PCIE_AT_IB_RP_BAR_ADDR0(bar), addr0);
> + cdns_pcie_writel(pcie, CDNS_PCIE_AT_IB_RP_BAR_ADDR1(bar), addr1);
> +}
> +
> +static int cdns_pcie_host_bar_config(struct cdns_pcie_rc *rc,
> + struct resource_entry *entry,
> + enum cdns_pcie_rp_bar *index)
> +{
> + u64 cpu_addr, pci_addr, size, winsize;
> + struct cdns_pcie *pcie = &rc->pcie;
> + struct device *dev = pcie->dev;
> + enum cdns_pcie_rp_bar bar;
> + unsigned long flags;
> + u32 aperture;
> + u32 value;
> +
> + cpu_addr = entry->res->start;
> + flags = entry->res->flags;
> + pci_addr = entry->res->start - entry->offset;
> + size = resource_size(entry->res);
> + bar = *index;
> +
> + if (entry->offset) {
> + dev_err(dev, "Cannot map PCI addr: %llx to CPU addr: %llx\n",
> + pci_addr, cpu_addr);
Would be a bit more clear to say PCI addr must equal CPU addr.
> + return -EINVAL;
> + }
> +
> + value = cdns_pcie_readl(pcie, CDNS_PCIE_LM_RC_BAR_CFG);
> + while (size > 0) {
> + if (bar > RP_NO_BAR) {
> + dev_err(dev, "Failed to map inbound regions!\n");
> + return -EINVAL;
> + }
> +
> + winsize = size;
> + if (size > cdns_rp_bar_max_size[bar])
> + winsize = cdns_rp_bar_max_size[bar];
> +
> + aperture = ilog2(winsize);
> +
> + cdns_pcie_host_bar_ib_config(rc, bar, cpu_addr, aperture);
> +
> + if (bar == RP_NO_BAR)
> + break;
> +
> + if (winsize + cpu_addr >= SZ_4G) {
> + if (!(flags & IORESOURCE_PREFETCH))
> + value |= LM_RC_BAR_CFG_CTRL_MEM_64BITS(bar);
> + value |= LM_RC_BAR_CFG_CTRL_PREF_MEM_64BITS(bar);
> + } else {
> + if (!(flags & IORESOURCE_PREFETCH))
> + value |= LM_RC_BAR_CFG_CTRL_MEM_32BITS(bar);
> + value |= LM_RC_BAR_CFG_CTRL_PREF_MEM_32BITS(bar);
> + }
> +
> + value |= LM_RC_BAR_CFG_APERTURE(bar, aperture);
> +
> + size -= winsize;
> + cpu_addr += winsize;
> + bar++;
> + }
> + cdns_pcie_writel(pcie, CDNS_PCIE_LM_RC_BAR_CFG, value);
> + *index = bar;
> +
> + return 0;
> +}
> +
> +static int cdns_pcie_host_map_dma_ranges(struct cdns_pcie_rc *rc)
> +{
> + enum cdns_pcie_rp_bar bar = RP_BAR0;
> + struct cdns_pcie *pcie = &rc->pcie;
> + struct device *dev = pcie->dev;
> + struct device_node *np = dev->of_node;
> + struct pci_host_bridge *bridge;
> + struct resource_entry *entry;
> + u32 no_bar_nbits = 32;
> + int err;
> +
> + bridge = pci_host_bridge_from_priv(rc);
> + if (!bridge)
> + return -ENOMEM;
> +
> + if (list_empty(&bridge->dma_ranges)) {
> + of_property_read_u32(np, "cdns,no-bar-match-nbits",
> + &no_bar_nbits);
> + cdns_pcie_host_bar_ib_config(rc, RP_NO_BAR, 0x0, no_bar_nbits);
> + return 0;
> + }
> +
> + resource_list_for_each_entry(entry, &bridge->dma_ranges) {
> + err = cdns_pcie_host_bar_config(rc, entry, &bar);
Seems like this should have some better logic to pick which BAR to
use. Something like find the biggest region and then find the smallest
BAR that it fits in. Then get the next biggest...
> + if (err) {
> + dev_err(dev, "Fail to configure IB using dma-ranges\n");
> + return err;
> + }
> + }
> +
> + return 0;
> +}
> +
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [EXT] Re: [PATCH net-next 3/5] net: mvpp2: cls: Use RSS contexts to handle RSS tables
From: Matteo Croce @ 2020-05-19 17:05 UTC (permalink / raw)
To: Russell King - ARM Linux admin
Cc: Antoine Tenart, netdev, gregory.clement@bootlin.com, LKML,
Maxime Chevallier, Nadav Haklai, Thomas Petazzoni,
miquel.raynal@bootlin.com, Stefan Chulski, Marcin Wojtas,
David S . Miller, Linux ARM
In-Reply-To: <CAGnkfhzuyxJDo-DXPHPiNtP4RbRpry+3M9eoiTknGR0zvgPuoA@mail.gmail.com>
On Tue, 19 May 2020 12:05:20 +0200
Matteo Croce <mcroce@redhat.com> wrote:
> On Tue, May 19, 2020 at 11:54 AM Russell King - ARM Linux admin
> <linux@armlinux.org.uk> wrote:
> >
> > On Sat, May 09, 2020 at 09:20:50PM +0100, Russell King - ARM Linux
> > admin wrote:
> > > On Sat, May 09, 2020 at 08:52:46PM +0100, Russell King - ARM
> > > Linux admin wrote:
> > > > It is highly likely that 895586d5dc32 is responsible for this
> > > > breakage. I've been investigating this afternoon, and what I've
> > > > found, comparing a kernel without 895586d5dc32 and with
> > > > 895586d5dc32 applied is:
> > > >
> > > > - The table programmed into the hardware via
> > > > mvpp22_rss_fill_table() appears to be identical with or without
> > > > the commit.
> > > >
> > > > - When rxhash is enabled on eth2, mvpp2_rss_port_c2_enable()
> > > > reports that c2.attr[0] and c2.attr[2] are written back
> > > > containing:
> > > >
> > > > - with 895586d5dc32, failing: 00200000 40000000
> > > > - without 895586d5dc32, working: 04000000 40000000
> > > >
> > > > - When disabling rxhash, c2.attr[0] and c2.attr[2] are written
> > > > back as:
> > > >
> > > > 04000000 00000000
> > > >
> > > > The second value represents the MVPP22_CLS_C2_ATTR2_RSS_EN bit,
> > > > the first value is the queue number, which comprises two
> > > > fields. The high 5 bits are 24:29 and the low three are 21:23
> > > > inclusive. This comes from:
> > > >
> > > > c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) |
> > > > MVPP22_CLS_C2_ATTR0_QLOW(ql);
> > > > #define MVPP22_CLS_C2_ATTR0_QHIGH(qh) (((qh) & 0x1f)
> > > > << 24) #define MVPP22_CLS_C2_ATTR0_QLOW(ql) (((ql) &
> > > > 0x7) << 21)
> > > >
> > > > So, the working case gives eth2 a queue id of 4.0, or 32 as per
> > > > port->first_rxq, and the non-working case a queue id of 0.1, or
> > > > 1.
> > > >
> > > > The allocation of queue IDs seems to be in mvpp2_port_probe():
> > > >
> > > > if (priv->hw_version == MVPP21)
> > > > port->first_rxq = port->id * port->nrxqs;
> > > > else
> > > > port->first_rxq = port->id *
> > > > priv->max_port_rxqs;
> > > >
> > > > Where:
> > > >
> > > > if (priv->hw_version == MVPP21)
> > > > priv->max_port_rxqs = 8;
> > > > else
> > > > priv->max_port_rxqs = 32;
> > > >
> > > > Making the port 0 (eth0 / eth1) have port->first_rxq = 0, and
> > > > port 1 (eth2) be 32. It seems the idea is that the first 32
> > > > queues belong to port 0, the second 32 queues belong to port 1,
> > > > etc.
> > > >
> > > > mvpp2_rss_port_c2_enable() gets the queue number from it's
> > > > parameter, 'ctx', which comes from mvpp22_rss_ctx(port, 0).
> > > > This returns port->rss_ctx[0].
> > > >
> > > > mvpp22_rss_context_create() is responsible for allocating that,
> > > > which it does by looking for an unallocated priv->rss_tables[]
> > > > pointer. This table is shared amongst all ports on the CP
> > > > silicon.
> > > >
> > > > When we write the tables in mvpp22_rss_fill_table(), the RSS
> > > > table entry is defined by:
> > > >
> > > > u32 sel = MVPP22_RSS_INDEX_TABLE(rss_ctx) |
> > > > MVPP22_RSS_INDEX_TABLE_ENTRY(i);
> > > >
> > > > where rss_ctx is the context ID (queue number) and i is the
> > > > index in the table.
> > > >
> > > > #define MVPP22_RSS_INDEX_TABLE_ENTRY(idx) (idx)
> > > > #define MVPP22_RSS_INDEX_TABLE(idx) ((idx) << 8)
> > > > #define MVPP22_RSS_INDEX_QUEUE(idx) ((idx) << 16)
> > > >
> > > > If we look at what is written:
> > > >
> > > > - The first table to be written has "sel" values of
> > > > 00000000..0000001f, containing values 0..3. This appears to be
> > > > for eth1. This is table 0, RX queue number 0.
> > > > - The second table has "sel" values of 00000100..0000011f, and
> > > > appears to be for eth2. These contain values 0x20..0x23. This
> > > > is table 1, RX queue number 0.
> > > > - The third table has "sel" values of 00000200..0000021f, and
> > > > appears to be for eth3. These contain values 0x40..0x43. This
> > > > is table 2, RX queue number 0.
> > > >
> > > > Okay, so how do queue numbers translate to the RSS table?
> > > > There is another table - the RXQ2RSS table, indexed by the
> > > > MVPP22_RSS_INDEX_QUEUE field of MVPP22_RSS_INDEX and accessed
> > > > through the MVPP22_RXQ2RSS_TABLE register. Before
> > > > 895586d5dc32, it was:
> > > >
> > > > mvpp2_write(priv, MVPP22_RSS_INDEX,
> > > > MVPP22_RSS_INDEX_QUEUE(port->first_rxq));
> > > > mvpp2_write(priv, MVPP22_RXQ2RSS_TABLE,
> > > > MVPP22_RSS_TABLE_POINTER(port->id));
> > > >
> > > > and after:
> > > >
> > > > mvpp2_write(priv, MVPP22_RSS_INDEX,
> > > > MVPP22_RSS_INDEX_QUEUE(ctx)); mvpp2_write(priv,
> > > > MVPP22_RXQ2RSS_TABLE, MVPP22_RSS_TABLE_POINTER(ctx));
> > > >
> > > > So, before the commit, for eth2, that would've contained '32'
> > > > for the index and '1' for the table pointer - mapping queue 32
> > > > to table 1. Remember that this is queue-high.queue-low of 4.0.
> > > >
> > > > After the commit, we appear to map queue 1 to table 1. That
> > > > again looks fine on the face of it.
> > > >
> > > > Section 9.3.1 of the A8040 manual seems indicate the reason
> > > > that the queue number is separated. queue-low seems to always
> > > > come from the classifier, whereas queue-high can be from the
> > > > ingress physical port number or the classifier depending on the
> > > > MVPP2_CLS_SWFWD_PCTRL_REG.
> > > >
> > > > We set the port bit in MVPP2_CLS_SWFWD_PCTRL_REG, meaning that
> > > > queue-high comes from the MVPP2_CLS_SWFWD_P2HQ_REG()
> > > > register... and this seems to be where our bug comes from.
> > > >
> > > > mvpp2_cls_oversize_rxq_set() sets this up as:
> > > >
> > > > mvpp2_write(port->priv,
> > > > MVPP2_CLS_SWFWD_P2HQ_REG(port->id), (port->first_rxq >>
> > > > MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS));
> > > >
> > > > val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG);
> > > > val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id);
> > > > mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);
> > > >
> > > > so, the queue-high for eth2 is _always_ 4, meaning that only
> > > > queues 32 through 39 inclusive are available to eth2. Yet,
> > > > we're trying to tell the classifier to set queue-high, which
> > > > will be ignored, to zero.
> > > >
> > > > So we end up directing traffic from eth2 not to queue 1, but to
> > > > queue 33, and then we tell it to look up queue 33 in the RSS
> > > > table. However, RSS table has not been programmed for queue
> > > > 33, and so it ends up (presumably) dropping the packets.
> > > >
> > > > It seems that mvpp22_rss_context_create() doesn't take account
> > > > of the fact that the upper 5 bits of the queue ID can't
> > > > actually be changed due to the settings in
> > > > mvpp2_cls_oversize_rxq_set(), _or_ it seems that
> > > > mvpp2_cls_oversize_rxq_set() has been missed in this commit.
> > > > Either way, these two functions mutually disagree with what
> > > > queue number should be used.
> > > >
> > > > So, 895586d5dc32 is indeed the cause of this problem.
> > >
> > > Looking deeper into what mvpp2_cls_oversize_rxq_set() and the MTU
> > > validation is doing, it seems that MVPP2_CLS_SWFWD_P2HQ_REG() is
> > > used for at least a couple of things.
> > >
> > > So, with the classifier having had RSS enabled and directing eth2
> > > traffic to queue 1, we can not ignore the fact that we may have
> > > packets appearing on queue 32 for this port.
> > >
> > > One of the things that queue 32 will be used for is if an
> > > over-sized packet attempts to egress through eth2 - it seems that
> > > the A8040 has the ability to forward frames between its ports.
> > > However, afaik we don't support that feature, and the kernel
> > > restricts the packet size, so we should never violate the MTU
> > > validator and end up with such a packet. In any case, _if_ we
> > > were to attempt to transmit an oversized packet, we have no
> > > support in the kernel to deal with that appearing in the port's
> > > receive queue.
> > >
> > > Maybe it would be safe to clear the MVPP2_CLS_SWFWD_PCTRL_MASK()
> > > bit?
> > >
> > > My testing seems to confirm my findings above - clearing this bit
> > > means that if I enable rxhash on eth2, the interface can then pass
> > > traffic, as we are now directing traffic to RX queue 1 rather than
> > > queue 33. Traffic still seems to work with rxhash off as well.
> > >
> > > So, I think it's clear where the problem lies, but not what the
> > > correct solution is; someone with more experience of packet
> > > classifiers (this one?) needs to look at this - this is my first
> > > venture into these things, and certainly the first time I've
> > > traced through how this is trying to work (or not)...
> >
> > This is what I was using here to work around the problem, and what I
> > mentioned above.
> >
> > diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c
> > b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c index
> > fd221d88811e..0dd3b65822dd 100644 ---
> > a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +++
> > b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c @@ -1058,7 +1058,7
> > @@ void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port)
> > (port->first_rxq >> MVPP2_CLS_OVERSIZE_RXQ_LOW_BITS));
> >
> > val = mvpp2_read(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG);
> > - val |= MVPP2_CLS_SWFWD_PCTRL_MASK(port->id);
> > + val &= ~MVPP2_CLS_SWFWD_PCTRL_MASK(port->id);
> > mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val);
> > }
> >
> >
>
> Hi,
>
> I will try this change and let you know if it works.
>
> Thanks
>
Hi,
The patch seems to work. I'm generating traffic with random MAC and IP
addresses, to have many flows:
# tcpdump -tenni eth2
9a:a9:b1:3a:b1:6b > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12
9e:92:fd:f8:7f:0a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.4.0 > 192.168.0.4.0: UDP, length 12
66:b7:11:8a:c2:1f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12
7a:ba:58:bd:9a:62 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.1.0 > 192.168.0.1.0: UDP, length 12
7e:78:a9:97:70:3a > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12
b2:81:91:34:ce:42 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.2.0 > 192.168.0.2.0: UDP, length 12
2a:05:52:d0:d9:3f > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12
ee:ee:47:35:fa:81 > 00:51:82:11:22:02, ethertype IPv4 (0x0800), length 60: 10.0.0.3.0 > 192.168.0.3.0: UDP, length 12
This is the default rate, with rxhash off:
# utraf eth2
tx: 0 bps 0 pps rx: 397.4 Mbps 827.9 Kpps
tx: 0 bps 0 pps rx: 396.3 Mbps 825.7 Kpps
tx: 0 bps 0 pps rx: 396.6 Mbps 826.3 Kpps
tx: 0 bps 0 pps rx: 396.5 Mbps 826.1 Kpps
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9 root 20 0 0 0 0 R 99.7 0.0 7:02.58 ksoftirqd/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
20 root 20 0 0 0 0 S 0.0 0.0 2:01.48 ksoftirqd/2
25 root 20 0 0 0 0 S 0.0 0.0 0:32.86 ksoftirqd/3
and this with rx hashing enabled:
# ethtool -K eth2 rxhash on
# utraf eth2
tx: 0 bps 0 pps rx: 456.4 Mbps 950.8 Kpps
tx: 0 bps 0 pps rx: 458.4 Mbps 955.0 Kpps
tx: 0 bps 0 pps rx: 457.6 Mbps 953.3 Kpps
tx: 0 bps 0 pps rx: 462.2 Mbps 962.9 Kpps
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 root 20 0 0 0 0 R 0.7 0.0 2:02.34 ksoftirqd/2
25 root 20 0 0 0 0 S 0.3 0.0 0:33.25 ksoftirqd/3
9 root 20 0 0 0 0 S 0.0 0.0 7:52.57 ksoftirqd/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
The throughput doesn't increase so much, maybe we hit an HW limit of
the gigabit port. The interesting thing is how the global CPU usage
drops from 25% to 1%.
I can't explain this, it could be due to the reduced contention?
Regards,
--
Matteo Croce
per aspera ad upstream
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH 2/2] ARM: Allow either FLATMEM or SPARSEMEM on the multiplatform build
From: Russell King - ARM Linux admin @ 2020-05-19 16:54 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Doug Berger, Stephen Boyd, Kevin Cernekee, Mike Rapoport,
Florian Fainelli, Gregory Fong, Linux ARM
In-Reply-To: <CAK8P3a2iZNm310x8g2Zv0TjGJ=Px7hu14i3Ka7GQBZwyKPUesA@mail.gmail.com>
On Tue, May 19, 2020 at 05:32:52PM +0200, Arnd Bergmann wrote:
> On Tue, May 19, 2020 at 5:27 PM Mike Rapoport <rppt@linux.ibm.com> wrote:
> > On Tue, May 19, 2020 at 04:04:52PM +0100, Russell King - ARM Linux admin wrote:
> > > There are cases where the reason to use sparsemem is not an optional
> > > one but is out of necessity - they require the page array to be split
> > > up in order to boot successfully.
> > >
> > > With that in mind, flatmem becomes an "optimisation" over sparsemem.
> >
> > At the moment, there are three platforms that enable SPARSEMEM: ARCH_EP93XX,
> > ARCH_RPC and ARCH_SA1100. All the rest have FLATMEM implcitly selected.
> >
> > I do not intend to change that, I am only going add an ability to select
> > the memory model for ARCH_MULTIPLATFORM.
> >
> > I'll respin the series on the list before adding the patches to the
> > patch system.
>
> I think we'll make EP93xx part of multiplatform at some point, IIRC
> only the missing clock driver is stopping us at the moment, and I already
> discussed with Linus Walleij how that can be done.
>
> My guess is that ep93xx is one platform on which sparsemem is
> just an optimization to reduce the initial memory consumption, but
> we should verify that when we get there.
When you have a platform where the memory is segmented into separate
blocks widely spaced, sparsemem is not an optimisation. For example,
If you have four blocks spaced across 1GB, that requires about
256Ki struct page's. Assuming 32 byte struct page, that requires 8MiB
of contiguous memory.
If we also consider the other constraint - that the kernel has to fit
in the first bank of memory, then, considering the size of the kernel
(the first two are non-multiplatform kernels):
text data bss dec hex filename
4045505 903700 92984 5042189 4cf00d n2100/boot/vmlinux-5.6.12+
4045361 957276 1159052 6161689 5e0519 assabet/boot/vmlinux-5.2.0+
6933363 1451420 203984 8588767 830ddf virt/boot/vmlinux-5.6.0+
9980260 3568070 7403296 20951626 13fb24a multi/boot/vmlinux-5.3.0+
So, realistically, we're looking at imposing a requirement that the
first bank of memory is no smaller than 16MB on these machines if a
"default" flatmem multiplatform kernel is going to be able to boot,
and if all banks are populated, that there is another bank that has
at least 8MB to hold the memmap array.
BTW, the "optimisation" argument for sparsemem doesn't actually
hold. For flatmem, we free the unused parts of the memmap array,
freeing those pages for other uses. Sparsemem on ARM is about
allowing these platforms to boot with their memory spread across
the physical address space savannah.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* Re: [PATCH] arch/{mips,sparc,microblaze,powerpc}: Don't enable pagefault/preempt twice
From: Guenter Roeck @ 2020-05-19 16:54 UTC (permalink / raw)
To: ira.weiny
Cc: Peter Zijlstra, Benjamin Herrenschmidt, Dave Hansen, dri-devel,
linux-mips, James E.J. Bottomley, Max Filippov, Paul Mackerras,
H. Peter Anvin, sparclinux, Dan Williams, Helge Deller, x86,
linux-csky, Christoph Hellwig, Ingo Molnar, linux-snps-arc,
linux-xtensa, Borislav Petkov, Al Viro, Andy Lutomirski,
Thomas Gleixner, linux-arm-kernel, Chris Zankel,
Thomas Bogendoerfer, linux-parisc, linux-kernel, Christian Koenig,
Andrew Morton, linuxppc-dev, David S. Miller
In-Reply-To: <20200518184843.3029640-1-ira.weiny@intel.com>
On Mon, May 18, 2020 at 11:48:43AM -0700, ira.weiny@intel.com wrote:
> From: Ira Weiny <ira.weiny@intel.com>
>
> The kunmap_atomic clean up failed to remove one set of pagefault/preempt
> enables when vaddr is not in the fixmap.
>
> Fixes: bee2128a09e6 ("arch/kunmap_atomic: consolidate duplicate code")
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
microblazeel works with this patch, as do the nosmp sparc32 boot tests,
but sparc32 boot tests with SMP enabled still fail with lots of messages
such as:
BUG: Bad page state in process swapper/0 pfn:006a1
page:f0933420 refcount:0 mapcount:1 mapping:(ptrval) index:0x1
flags: 0x0()
raw: 00000000 00000100 00000122 00000000 00000001 00000000 00000000 00000000
page dumped because: nonzero mapcount
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 5.7.0-rc6-next-20200518-00002-gb178d2d56f29 #1
[f00e7ab8 :
bad_page+0xa8/0x108 ]
[f00e8b54 :
free_pcppages_bulk+0x154/0x52c ]
[f00ea024 :
free_unref_page+0x54/0x6c ]
[f00ed864 :
free_reserved_area+0x58/0xec ]
[f0527104 :
kernel_init+0x14/0x110 ]
[f000b77c :
ret_from_kernel_thread+0xc/0x38 ]
[00000000 :
0x0 ]
Code path leading to that message is different but always the same
from free_unref_page().
Still testing ppc images.
Guenter
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox