gettimeofday scalability

gettimeofday scalability

[P]

I'm starting to look again at the performance of my packet sniffer.
Any performace tips are appreciated (I'm using irq affinity and
CONFIG_PACKET_MMAP on 2.4.20 on a dual P4 xeon at present).

In particular I was wondering about reducing the overhead of
calling do_gettimeofday.

I noticed in the following paper that the xeon is much less
efficient than the P3 for gettimeofday (for the syscall at least):
http://www.labs.fujitsu.com/en/techinfo/linux/lse-0211/lse-0211.pdf

I've seen various gettimeofday locking speedup patches floating
around for 2.4. There is a version from Stephen and Andrea
that uses frlock, claiming 18%, and one from ingo that uses brlock.
2.6.8.1 uses seqlock, which contains the comment
that it's not as cache friendly as brlock.

So can anyone summarise the relative merits of these locking
mechanisms, before I start benchmarking?

thanks,
Pádraig.

Re: gettimeofday scalability

[David S. Miller]

On Tue, 05 Oct 2004 17:36:06 +0100
P@draigBrady.com wrote:

> So can anyone summarise the relative merits of these locking
> mechanisms, before I start benchmarking?

Seq locks plus the timer interpolator layer found in 2.6.x kernels
is the most scalable gettimeofday() implementation currently.

Re: gettimeofday scalability

[Stephen Hemminger]

On Tue, 2004-10-05 at 17:36 +0100, P@draigBrady.com wrote:
> I'm starting to look again at the performance of my packet sniffer.
> Any performace tips are appreciated (I'm using irq affinity and
> CONFIG_PACKET_MMAP on 2.4.20 on a dual P4 xeon at present).
> 
> In particular I was wondering about reducing the overhead of
> calling do_gettimeofday.
> 
> I noticed in the following paper that the xeon is much less
> efficient than the P3 for gettimeofday (for the syscall at least):
> http://www.labs.fujitsu.com/en/techinfo/linux/lse-0211/lse-0211.pdf
> 
> I've seen various gettimeofday locking speedup patches floating
> around for 2.4. There is a version from Stephen and Andrea
> that uses frlock, claiming 18%, and one from ingo that uses brlock.
> 2.6.8.1 uses seqlock, which contains the comment
> that it's not as cache friendly as brlock.

Don't bother with doing new work on 2.4. Look at 2.6.
You could use TSC in user space but you aren't going to see absolute
times and you run into all the portablity, and possible speed change
issues.

> So can anyone summarise the relative merits of these locking
> mechanisms, before I start benchmarking?
> 
> thanks,
> Pádraig.

Re: gettimeofday scalability

[Andrea Arcangeli]

On Tue, Oct 05, 2004 at 05:36:06PM +0100, P@draigBrady.com wrote:
> I've seen various gettimeofday locking speedup patches floating
> around for 2.4. There is a version from Stephen and Andrea
> that uses frlock, claiming 18%, and one from ingo that uses brlock.
> 2.6.8.1 uses seqlock, which contains the comment
> that it's not as cache friendly as brlock.

seqlock and frlock are the same thing. I don't see how the brlock can
work well given the fact you'll have to take it in write mode at every
timer irq. Maybe it works on a 2-way, sure not more than that. brlock
should be totally replaced by RCU anyways. brlock can also starve the
writer, which make it a security DoS (at least for some architecture,
there were two implementations, maybe one is safe).

> So can anyone summarise the relative merits of these locking
> mechanisms, before I start benchmarking?

frlock/seqlock (2.4/2.6 respectively) is the way to go, no write
starvation, and zero cacheline bouncing. 

upgrade to x86-64, there I implemented gettimeofday with vsyscalls which
also avoids entering exiting userspace which becomes the by far biggest
overhead after using seqlock. (speedup is tenfold or so)

Hope this helps.

Re: gettimeofday scalability

[P]

Andrea Arcangeli wrote:
> On Tue, Oct 05, 2004 at 05:36:06PM +0100, P@draigBrady.com wrote:
> 
>>So can anyone summarise the relative merits of these locking
>>mechanisms, before I start benchmarking?
> 
> 
> frlock/seqlock (2.4/2.6 respectively) is the way to go, no write
> starvation, and zero cacheline bouncing. 

Cheers.

Perhaps the confusing comment wrt brlock at the
top of seqlock.h can be changed so?

> upgrade to x86-64, there I implemented gettimeofday with vsyscalls which
> also avoids entering exiting userspace which becomes the by far biggest
> overhead after using seqlock. (speedup is tenfold or so)

This is all in kernel space.
However Stephen's suggestion of reading the tsc in user space
may be a runner, as I just care about relative times.

thanks guys!

Pádraig.

Re: gettimeofday scalability

[Andrea Arcangeli]

On Tue, Oct 05, 2004 at 08:16:33PM +0100, P@draigBrady.com wrote:
> Andrea Arcangeli wrote:
> >On Tue, Oct 05, 2004 at 05:36:06PM +0100, P@draigBrady.com wrote:
> >
> >>So can anyone summarise the relative merits of these locking
> >>mechanisms, before I start benchmarking?
> >
> >
> >frlock/seqlock (2.4/2.6 respectively) is the way to go, no write
> >starvation, and zero cacheline bouncing. 
> 
> Cheers.
> 
> Perhaps the confusing comment wrt brlock at the
> top of seqlock.h can be changed so?

I guess so.

> This is all in kernel space.

vsyscalls are in userspace, but you will not notice the difference.

Or do you mean your code is in kernel space? vsyscalls would run from
kernel space too, but then you can use gettimeofday by hand with seqlock
and it won't be any different.

> However Stephen's suggestion of reading the tsc in user space
> may be a runner, as I just care about relative times.

Stephen's suggestion will lead to your app breaking on asymmetric TSC on
SMP if you're not careful, if you use the vsyscall on a correct kernel
(like x86-64) you won't take that risk (HPETS avoids that, slower than
the tsc but the only safe one). Otherwise you've to use process affinity
+ tsc, only with cpu binding you're safe. Some big app uses TSC but only
optionally, so you can turn it on/off depending on the hardware (on
x86-64 this obsoleted by vgettimeofday, that's a x86-only hack).

hope this helps ;)

Re: gettimeofday scalability

[Ingo Molnar]

* P@draigBrady.com <P@draigBrady.com> wrote:

> In particular I was wondering about reducing the overhead of
> calling do_gettimeofday.

> 2.6.8.1 uses seqlock, which contains the comment that it's not as
> cache friendly as brlock.

that comment is way too modest! Seqlocks are very cache-friendly in the
read path. There is no reason to use brlocks anymore for fixed-frequency
writers like the timer seqlock. (writers can starve seqlock readers but
in the timer case the writers occur only once every 1 msec.)

so please benchmark 2.6, it should scale linearly in this area.

	Ingo