From: Dave Chinner <david@fromorbit.com>
To: Nick Piggin <npiggin@kernel.dk>
Cc: xfs@oss.sgi.com
Subject: Re: XFS performance oddity
Date: Wed, 24 Nov 2010 14:15:24 +1100 [thread overview]
Message-ID: <20101124031524.GC22876@dastard> (raw)
In-Reply-To: <20101124005003.GB3168@amd>
On Wed, Nov 24, 2010 at 11:50:03AM +1100, Nick Piggin wrote:
> On Wed, Nov 24, 2010 at 07:58:04AM +1100, Dave Chinner wrote:
> > On Tue, Nov 23, 2010 at 11:24:49PM +1100, Nick Piggin wrote:
> > > Hi,
> > >
> > > Running parallel fs_mark (0 size inodes, fsync on close) on a ramdisk
> > > ends up with XFS in funny patterns.
> > >
> > > procs -----------memory---------- ---swap-- -----io---- -system--
> > > ----cpu----
> > > r b swpd free buff cache si so bi bo in cs us sy
> > > id wa
> > > 24 1 6576 166396 252 393676 132 140 16900 80666 21308 104333 1 84 14 1
> > > 21 0 6712 433856 256 387080 100 224 9152 53487 13677 53732 0 55 45 0
> > > 2 0 7068 463496 248 389100 0 364 2940 17896 4485 26122 0 33 65 2
> > > 1 0 7068 464340 248 388928 0 0 0 0 66 207 0 0 100 0
> > > 0 0 7068 464340 248 388928 0 0 0 0 79 200 0 0 100 0
> > > 0 0 7068 464544 248 388928 0 0 0 0 65 199 0 0 100 0
> > > 1 0 7068 464748 248 388928 0 0 0 0 79 201 0 0 100 0
> > > 0 0 7068 465064 248 388928 0 0 0 0 66 202 0 0 100 0
> > > 0 0 7068 465312 248 388928 0 0 0 0 80 200 0 0 100 0
> > > 0 0 7068 465500 248 388928 0 0 0 0 65 199 0 0 100 0
> > > 0 0 7068 465500 248 388928 0 0 0 0 80 202 0 0 100 0
> > > 1 0 7068 465500 248 388928 0 0 0 0 66 203 0 0 100 0
> > > 0 0 7068 465500 248 388928 0 0 0 0 79 200 0 0 100 0
> > > 23 0 7068 460332 248 388800 0 0 1416 8896 1981 7142 0 1 99 0
> > > 6 0 6968 360248 248 403736 56 0 15568 95171 19438 110825 1 79 21 0
> > > 23 0 6904 248736 248 419704 392 0 17412 118270 20208 111396 1 82 17 0
> > > 9 0 6884 266116 248 435904 128 0 14956 79756 18554 118020 1 76 23 0
> > > 0 0 6848 219640 248 445760 212 0 9932 51572 12622 76491 0 60 40 0
> > >
> > > Got a dump of sleeping tasks. Any ideas?
> >
> > It is stuck waiting for log space to be freed up. Generally this is
> > caused by log IO completion not occurring or an unflushable object
> > preventing the tail from being moved forward. What:
>
> Yeah it's strage, it seems like it hits some timeout or gets kicked
> along by background writeback or something. Missed wakeup somewhere?
No idea yet.
> > - is the output of mkfs.xfs?
>
> meta-data=/dev/ram0 isize=256 agcount=16, agsize=65536 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=1048576, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal log bsize=4096 blocks=16384, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
Ok, small log, small AGs.
> > - are your mount options?
>
> mount -o delaylog,logbsize=262144,nobarrier /dev/ram0 mnt
>
> > - is the fs_mark command line?
>
> ../fs_mark -S1 -k -n 1000 -L 100 -s 0 -d scratch/0 -d scratch/1 -d
> scratch/2 -d scratch/3 -d scratch/4 -d scratch/5 -d scratch/6 -d
> scratch/7 -d scratch/8 -d scratch/9 -d scratch/10 -d scratch/11 -d
> scratch/12 -d scratch/13 -d scratch/14 -d scratch/15 -d scratch/16 -d
> scratch/17 -d scratch/18 -d scratch/19 -d scratch/20 -d scratch/21 -d
> scratch/22 -d scratch/23
> for f in scratch/* ; do rm -rf $f & done ; wait
Ok, so you are effectively doing a concurrent synchronous create of 2.4M
zero byte files.
BTW, how many CPU cores does your machine have? if it's more than 8, then
you're probably getting a fair bit of serialisation on the per-ag
structures. I normally use agcount=num_cpus * 2 for scalability
testing when running one load thread per CPU.
> Ran it again, and yes it has locked up for a long long time, it seems
> to be in the rm phase, but I think I've seen similar stall (although not
> so long) in the fs_mark phase too.
Ok, I've just reproduced a couple of short hangs (a few seconds)
during the rm phase so I should be able to track it down.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2010-11-24 3:13 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-23 12:24 XFS performance oddity Nick Piggin
2010-11-23 12:30 ` Nick Piggin
2010-11-23 20:58 ` Dave Chinner
2010-11-24 0:50 ` Nick Piggin
2010-11-24 3:15 ` Dave Chinner [this message]
2010-11-24 5:28 ` [PATCH] " Dave Chinner
2010-11-25 4:54 ` Nick Piggin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20101124031524.GC22876@dastard \
--to=david@fromorbit.com \
--cc=npiggin@kernel.dk \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.