Re: Unacceptably Poor RAID1 Performance with Many CPU Cores

[Ali Gholami Rudi <aligrudi@gmail.com>]

Hi Xiao,

Xiao Ni <xni@redhat.com> wrote:
> > Ali Gholami Rudi <aligrudi@gmail.com> wrote:
> > > Xiao Ni <xni@redhat.com> wrote:
> > > > Because it can be reproduced easily in your environment. Can you try
> > > > with the latest upstream kernel? If the problem doesn't exist with
> > > > latest upstream kernel. You can use git bisect to find which patch can
> > > > fix this problem.
> > >
> > > I just tried the upstream.  I get almost the same result with 1G ramdisks.
> > >
> > > Without RAID (writing to /dev/ram0)
> > > READ:  IOPS=15.8M BW=60.3GiB/s
> > > WRITE: IOPS= 6.8M BW=27.7GiB/s
> > >
> > > RAID1 (writing to /dev/md/test)
> > > READ:  IOPS=518K BW=2028MiB/s
> > > WRITE: IOPS=222K BW= 912MiB/s
> 
> I can reproduce this with upstream kernel too.
> 
> RAID1
> READ: bw=3699MiB/s (3879MB/s)
> WRITE: bw=1586MiB/s (1663MB/s)
> 
> ram disk:
> READ: bw=5720MiB/s (5997MB/s)
> WRITE: bw=2451MiB/s (2570MB/s)
> 
> There is a performance problem. But not like your result. Your result
> has a huge gap. I'm not sure the reason. Any thoughts?

It may be the number of cores; in my setup there are 128 cores (256
threads).  If I understand it correctly, the problem is that
wakeup(...->wait_barrier) is called unconditionally at the same time
by many cores in functions like allow_barrier and flush_bio_list,
while no task is waiting in the task queue.  This results in a
lock contention for wq_head->lock in __wakekup_common_lock().

> > And this is perf's output:
> 
> I'm not familiar with perf, what's your command that I can use to see
> the same output?

perf record --call-graph dwarf fio fio.test

perf report

Thanks,
Ali