Re: [PATCH V3 11/14] coresight: sink: Add TRBE driver

Anshuman Khandual
2021-02-12T05:44:16Z

On 2/11/21 12:30 AM, Mathieu Poirier wrote:
> On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote:
>> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is
>> accessible via the system registers. The TRBE supports different addressing
>> modes including CPU virtual address and buffer modes including the circular
>> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1),
>> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the
>> access to the trace buffer could be prohibited by a higher exception level
>> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU
>> private interrupt (PPI) on address translation errors and when the buffer
>> is full. Overall implementation here is inspired from the Arm SPE driver.
>>
>> Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
>> Cc: Mike Leach <mike.leach@linaro.org>
>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>> ---
>> Changes in V3:
>>
>> - Added new DT bindings document TRBE.yaml
>> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3
>> - Dropped isb() from trbe_reset_local()
>> - Dropped gap between (void *) and buf->trbe_base
>> - Changed 'int' to 'unsigned int' in is_trbe_available()
>> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(),
>> set_trbe_enabled() and set_trbe_limit_pointer()
>> - Changed get_trbe_flag_update(), is_trbe_programmable() and
>> get_trbe_address_align() to accept TRBIDR value
>> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(),
>> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value
>> - Dropped snapshot mode condition in arm_trbe_alloc_buffer()
>> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled
>> - Compute trbe_limit before trbe_write to get the updated handle
>> - Added trbe_stop_and_truncate_event()
>> - Dropped trbe_handle_fatal()
>>
>> Documentation/trace/coresight/coresight-trbe.rst | 39 +
>> arch/arm64/include/asm/sysreg.h | 1 +
>> drivers/hwtracing/coresight/Kconfig | 11 +
>> drivers/hwtracing/coresight/Makefile | 1 +
>> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++
>> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++
>> 6 files changed, 1235 insertions(+)
>> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst
>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c
>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h
>> +
>
> [...]
>
>> +static void arm_trbe_probe_coresight_cpu(void *info)
>> +{
>> + struct trbe_drvdata *drvdata = info;
>> + struct coresight_desc desc = { 0 };
>> + int cpu = smp_processor_id();
>> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu);
>> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu);
>> + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1);
>> + struct device *dev;
>> +
>> + if (WARN_ON(!cpudata))
>> + goto cpu_clear;
>
> There is already a check for this in arm_trbe_probe_coresight(), we couldn't be
> here if there was a problem with the allocation.

Right but just to be extra cautious. Do you really want this to be dropped ?

>
>> +
>> + if (trbe_csdev)
>> + return;
>
> Now that's a reason to have a WARN_ON(). If we are probing and a sink is
> already present in this cpu's slot, something went seriously wrong and we should
> be clear about it.

Right, will add an WARN_ON().

>
>> +
>> + cpudata->cpu = smp_processor_id();
>> + cpudata->drvdata = drvdata;
>> + dev = &cpudata->drvdata->pdev->dev;
>> +
>> + if (!is_trbe_available()) {
>> + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu);
>> + goto cpu_clear;
>> + }
>> +
>> + if (!is_trbe_programmable(trbidr)) {
>> + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu);
>> + goto cpu_clear;
>> + }
>> + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id());
>
> We will end up with "arm_trbe0", "arm_trbe1" and so on in sysfs... Is the
> "arm_" part absolutely needed? I think this should be like what we do for etmv3
> and etmv4 where only "etmX" shows up in sysfs.

Okay, will drop arm_ here. IIRC this was originally trbeX where X is the cpu number
but then ended up using DRVNAME as prefix.

>
>> + if (IS_ERR(desc.name))
>> + goto cpu_clear;
>> +
>> + desc.type = CORESIGHT_DEV_TYPE_SINK;
>> + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM;
>> + desc.ops = &arm_trbe_cs_ops;
>> + desc.pdata = dev_get_platdata(dev);
>> + desc.groups = arm_trbe_groups;
>> + desc.dev = dev;
>> + trbe_csdev = coresight_register(&desc);
>> + if (IS_ERR(trbe_csdev))
>> + goto cpu_clear;
>> +
>> + dev_set_drvdata(&trbe_csdev->dev, cpudata);
>> + cpudata->trbe_dbm = get_trbe_flag_update(trbidr);
>> + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr);
>> + if (cpudata->trbe_align > SZ_2K) {
>> + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu);
>> + goto cpu_clear;
>
> Here coresight_unregister() should be called. The other option is to call
> coresight_register() when everything else is known to be fine, which is the
> favoured approach.

Okay, will change accordingly.

>
>> + }
>> + per_cpu(csdev_sink, cpu) = trbe_csdev;
>> + trbe_reset_local();
>> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE);
>> + return;
>> +cpu_clear:
>> + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus);
>> +}
>> +
>> +static void arm_trbe_remove_coresight_cpu(void *info)
>> +{
>> + int cpu = smp_processor_id();
>> + struct trbe_drvdata *drvdata = info;
>> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu);
>> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu);
>> +
>> + if (trbe_csdev) {
>
> In what scenario do you see not having a trbe_csdev and still needing to disable
> IRQs for the HW? If there is a such a case then a few lines of comment is
> needed.
>
>> + coresight_unregister(trbe_csdev);
>> + cpudata->drvdata = NULL;
>> + per_cpu(csdev_sink, cpu) = NULL;
>> + }
>> + disable_percpu_irq(drvdata->irq);
>> + trbe_reset_local();
>
> Theoretically this code shouldn't run when the TRBE is enabled, because the CS
> core will prevent that from happening. As sush disabling interrupts after
> coresight_unregister() has been called and setting cpudata->drvdata to NULL
> should be fine. But from an outsider's point of view it will look very bizarre.
> Either write a comment to explain all that or call the above two before doing
> the cleanup.

Okay, will move them before the cleanup.

>
>> +}
>> +
>> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata)
>> +{
>> + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata));
>> + if (IS_ERR(drvdata->cpudata))
>> + return PTR_ERR(drvdata->cpudata);
>
> As far as I can tell alloc_percpu() returns NULL on failure and nothing else.

Sure, will change the return code as -ENOMEM when alloc_percpu() returns NULL.

>
>> +
>> + arm_trbe_probe_coresight_cpu(drvdata);
>> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1);
>
> The above two calls look racy to me. The executing process could be moved to
> another CPU between the call to arm_trbe_probe_coresight_cpu() and
> smp_call_function_many(), which would prevent the initialisation of the TRBE on
> the new CPU to be done. I suggest using a for_each_cpu() loop where
> smp_call_function_single() would be used. That way we are guaranteed all the
> TRBEs will be initialised.

Okay, will change.

>
>> + return 0;
>> +}
>> +
>> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata)
>> +{
>> + arm_trbe_remove_coresight_cpu(drvdata);
>> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1);
>
> Same as above.

Okay, will do.

>
> I'm out of time for today, more to come tomorrow.

Okay.

>
> Mathieu

- Anshuman