Beginner's questions on userspace latency

Beginner's questions on userspace latency

[Oliver Nittka]

Hello,

I need to get data from a data acquisition board into user space at
least every 2 milliseconds.

My setup: There's a kernel module that gets measurement data from a data
acquisition board via USB. We're talking about 6 channels @ 100kHz each,
so 512 bytes every 500us. Kernel timing is fine, I checked this using
get_cycles() and computing the delta to the previous urb callback.

The kernel module puts the data into a ring buffer which the user space
application accesses via mmap() (using remap_vmalloc_range()) which i
suppose has better timing compared to copy_to_user()?

Now the user space application needs to be notified at least every 2
milliseconds (so it can evaluate the data and trigger an action in due
time). Currently, I'm using an ioctl which calls
wait_event_interruptible(), the module then wakes up the waitq in its
urb_callback using wake_up_interruptible().

To minimize latency in user space, the application does the following
right at the beginning of its main():

  mlockall(MCL_FUTURE);

  schparm.sched_priority = 99;
  sched_setscheduler(0, SCHED_FIFO, &schparm);

  CPU_SET(1, &cpuset);
  sched_setaffinity(0, sizeof(cpuset), &cpuset);

I'm using a vanilla kernel 2.6.33.9 with patch-2.6.33.9-rt31.

I've also tuned the scheduler using
echo 500000 > /proc/sys/kernel/sched_latency_ns
echo 200000 > /proc/sys/kernel/sched_wakeup_granularity_ns
echo 100000 > /proc/sys/kernel/sched_min_granularity_ns

The timing is fine most of the time, but putting the system under heavy
CPU and USB load leads to occasional glitches where the time between two
ioctl calls can be as high as 50ms.

I'm an experienced developer on linux, but mostly on the application
front. Writing the device driver was as close to the hardware as I ever
got, yet ;-)

I read through the RT wiki, but I'm still unsure on how to proceed from
here. Is there any way to determine what's causing the glitches?
Anything I could do to improve the realtime behavior of my application?

Thank you all very much in advance!

O. Nittka

RE: Beginner's questions on userspace latency

[Staffan Tjernstrom]

Have you turned off all CPU power saving / turbo mode settings in the BIOS/UEFI ? 

-----Original Message-----
From: linux-rt-users-owner@vger.kernel.org [mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of Oliver Nittka
Sent: Friday, March 22, 2013 2:36 PM
To: RT
Subject: Beginner's questions on userspace latency

Hello,

I need to get data from a data acquisition board into user space at least every 2 milliseconds.

My setup: There's a kernel module that gets measurement data from a data acquisition board via USB. We're talking about 6 channels @ 100kHz each, so 512 bytes every 500us. Kernel timing is fine, I checked this using
get_cycles() and computing the delta to the previous urb callback.

The kernel module puts the data into a ring buffer which the user space application accesses via mmap() (using remap_vmalloc_range()) which i suppose has better timing compared to copy_to_user()?

Now the user space application needs to be notified at least every 2 milliseconds (so it can evaluate the data and trigger an action in due time). Currently, I'm using an ioctl which calls wait_event_interruptible(), the module then wakes up the waitq in its urb_callback using wake_up_interruptible().

To minimize latency in user space, the application does the following right at the beginning of its main():

  mlockall(MCL_FUTURE);

  schparm.sched_priority = 99;
  sched_setscheduler(0, SCHED_FIFO, &schparm);

  CPU_SET(1, &cpuset);
  sched_setaffinity(0, sizeof(cpuset), &cpuset);

I'm using a vanilla kernel 2.6.33.9 with patch-2.6.33.9-rt31.

I've also tuned the scheduler using
echo 500000 > /proc/sys/kernel/sched_latency_ns echo 200000 > /proc/sys/kernel/sched_wakeup_granularity_ns
echo 100000 > /proc/sys/kernel/sched_min_granularity_ns

The timing is fine most of the time, but putting the system under heavy CPU and USB load leads to occasional glitches where the time between two ioctl calls can be as high as 50ms.

I'm an experienced developer on linux, but mostly on the application front. Writing the device driver was as close to the hardware as I ever got, yet ;-)

I read through the RT wiki, but I'm still unsure on how to proceed from here. Is there any way to determine what's causing the glitches?
Anything I could do to improve the realtime behavior of my application?

Thank you all very much in advance!

O. Nittka



--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Beginner's questions on userspace latency

[Tim Sander]

Hi Oliver
> I read through the RT wiki, but I'm still unsure on how to proceed from
> here. Is there any way to determine what's causing the glitches?
> Anything I could do to improve the realtime behavior of my application?
Realtime has a lot to do with your computing hardware + the drivers. The worst 
driver determines your max latency. As you have not told on what hardware you 
are running it is hard to give some hints... other than do latency tracing if 
your hw is powerful enough. Also you might look at the nice latency charts 
from osadl.org. Find a similar system to yours and look at the  worst 
latencies recorded.

Also keep in mind that latency tracing needs some time. There are some 
recordings of talks available for example of Frank Rowand about latency 
tracing but they are probably a little bit dated as the don't acount to much 
for the much improved latency tracing infrastructure within the kernel.

Good luck
Tim

Re: Beginner's questions on userspace latency

[Thomas Gleixner]

On Fri, 22 Mar 2013, Oliver Nittka wrote:
> The timing is fine most of the time, but putting the system under heavy
> CPU and USB load leads to occasional glitches where the time between two
> ioctl calls can be as high as 50ms.

The kernel tracer is your friend. Enable the kernel function tracer
and add a check into your application which detects the issue and then
stops the trace. See the cyclictest source code for an example how to
do that.

Thanks,

	tglx

Re: Beginner's questions on userspace latency

[Oliver Nittka]

Thank you all very much for your insights!

I was experimenting some more on the weekend, and before I could
investigate everything I learned from you, CPU isolation did the trick
for me (using cset just like shown in the wiki).

I originally thought sched_setaffinity() would suffice, but giving the
application its own CPU helped a lot. Didn't get anything over 1.735 ms
for two days now.

FYI: this is running on a Kontron pITX-SP
http://de.kontron.com/products/boards+and+mezzanines/embedded+sbc/pitx+25+sbc/pitxsp.html.

Thanks again!

  -- o