Re: Unexplained variance in run-time of simple program (part 2)

[Marc Gonzalez <marc.w.gonzalez@free.fr>]

Hello Doctor Bandwidth,

I was secretly hoping you would chime in! :)

You've moved to Spain if I understand correctly?

On 07/04/2026 10:37, John D. McCalpin wrote:

> Going back to the example discussed at [in 2025-09], it looks like
> the benchmark takes ~2500 ns and that you are executing the
> benchmark 2^16 times.   Does that mean you are launching a new
> process 2^16 times?  If so, exactly what mechanism are you using?
> The variation you are reporting of 0.1 to 0.3 microseconds per
> execution seems very small to me.  Are you sure that the variation
> is in the "execution time", or could it be associated with the
> launching of the execution?   If the benchmark code is being
> executed 2^16 times in a loop in a single execution, then the
> situation quite different and performance counters should be very
> effective at determining the cause of the variability.


I understand that process creation is a costly operation.
I do run the benchmark several times from the same process.
(I plan on revisiting your above remarks after I digest them.)


> For the performance counter side of things, I would recommend
> programming the performance counters externally to the benchmark
> process and using inline RDPMC instructions to get the counter
> values (inside the benchmark executable).


Why do you suggest programming the PMCs from outside the benchmark process?
(Is it perhaps because it's simpler than using the kernel API?)

It never occurred to me that I could read the PMCs from user-space!

This is what I've been doing until now:

int open_event(u64 type, u64 config, int fd)
{
	struct perf_event_attr attr = {
		.type = type,
		.size = sizeof(attr),
		.config = config,
		.read_format = PERF_FORMAT_GROUP,
	};

	return syscall(SYS_perf_event_open, &attr, 0, 3, fd, 0);
}

	int main_fd = open_event(PERF_TYPE_HARDWARE, PERF_COUNT_HW_CPU_CYCLES, -1);
	open_event(PERF_TYPE_HARDWARE, PERF_COUNT_HW_INSTRUCTIONS, main_fd);
	open_event(PERF_TYPE_RAW, UOPS_EXECUTED, main_fd);
	open_event(PERF_TYPE_RAW, EXEC_STALLS, main_fd);

	for (int i = 0; i < 1000000; ++i)
	{
		ioctl(main_fd, PERF_EVENT_IOC_RESET, PERF_IOC_FLAG_GROUP);
		for (int i = 0; i < N; ++i) spin(ctx);
		if (read(main_fd, v, sizeof v) < sizeof v) return 2;
		printf("%lu %lu %lu %lu\n", v[1]/N, v[2]/N, v[3]/N, v[4]/N);
	}

The problem with that technique is that the PMCs are "polluted" by
the exit from ioctl & the entry into read.

Reading the PMCs from user-space with RDPMC solves this issue!

Found this interesting suggestion from 10 years ago:
https://community.intel.com/t5/Software-Tuning-Performance/How-to-read-performance-counters-by-rdpmc-instruction/m-p/1009043
I think you might be familiar with the responder ;)


> I am not certain what the lowest overhead mechanism might be for
> getting those performance counter values out of the program — I
> would probably try attaching to a persistent System V shared memory
> segment and simply storing the values in memory for later post-processing.


Are you implying that simply using printf might disturb the caches
from the write calls? (I redirect the output to a tmpfs.)


> On Intel SKX processors I measured the overhead of an RDPMC
> instruction at as low as ~25 cycles, with RDTSCP instructions taking
> a little bit longer (~40 cycles), but most importantly inline RDPMC
> does not involve any uncontrolled code paths or any hardware
> accesses outside the core.  RDTSCP is similar, but might be
> accessing off-core resources — the timing depends on both the core
> and uncore clock frequencies.

Thanks for the great suggestion. I'll be reading up on RDPMC.

Why do you think I would need RDTSCP?
At the moment, I just pin all cores at 2 GHz & count cycles using PMCs.

Reference for myself:
https://www.codestudy.net/blog/difference-between-rdtscp-rdtsc-memory-and-cpuid-rdtsc/

Regards