Re: 20% performance drop on PostgreSQL 9.2 from kernel 3.5.3 to 3.6-rc5 on AMD chipsets - bisected

Mike Galbraith
2012-09-15T14:47:45Z

On Sat, 2012-09-15 at 12:44 +0200, Borislav Petkov wrote:
> On Sat, Sep 15, 2012 at 06:11:02AM +0200, Mike Galbraith wrote:
> > My wild (and only) theory is that this is userspace spinlock related.
> > If so, starting the server and benchmark SCHED_BATCH should not only
> > kill the regression, but likely improve throughput as well.
>
> FWIW,
>
> I went and tried it. Here are the exact steps:
>
> $ ps ax | grep postgres
> 2066 ? S 0:01 /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
> 2070 ? Ss 0:07 postgres: writer process
> 2071 ? Ss 0:05 postgres: wal writer process
> 2072 ? Ss 0:01 postgres: autovacuum launcher process
> 2073 ? Ss 0:01 postgres: stats collector process
> 5788 pts/0 S+ 0:00 grep postgres
>
> # set to SCHED_BATCH
> $ schedtool -B 2066 2070 2071 2072 2073
>
> # verify:
> $ schedtool 2066 2070 2071 2072 2073
> PID 2066: PRIO 0, POLICY B: SCHED_BATCH , NICE 0, AFFINITY 0x3f
> PID 2070: PRIO 0, POLICY B: SCHED_BATCH , NICE 0, AFFINITY 0x3f
> PID 2071: PRIO 0, POLICY B: SCHED_BATCH , NICE 0, AFFINITY 0x3f
> PID 2072: PRIO 0, POLICY B: SCHED_BATCH , NICE 0, AFFINITY 0x3f
> PID 2073: PRIO 0, POLICY B: SCHED_BATCH , NICE 0, AFFINITY 0x3f
>
> $ su - postgres
> postgres@hhost:~$ export PATH=$PATH:/usr/lib/postgresql/9.1/bin/
> postgres@hhost:~$ schedtool -B -e pgbench -i pgbench && pgbench -c 10 -t 10000 pgbench
>
> ...
>
> tps = 4388.118940 (including connections establishing)
> tps = 4391.771875 (excluding connections establishing)
>
> => even better than the results with 3.5 (had something around 3900ish
> on that particular configuration).

Increasing /proc/sys/kernel/sched_min_granularity_ns to roughly half of
sched_latency_ns should also help. That will allow LAST_BUDDY to do
it's job, try to hand the CPU back to a preempted task if possible. The
change that increased sched_nr_latency to 8 should have injured
postgress as well. ATM, it's disabled unless you're massively loaded.

I _think_ it's about preemption, but it doesn't matter, patch is toast.

-Mike