XFS mkfs/mount options

XFS mkfs/mount options

[Mark]

I have been doing some experiments with XFS on my "hot new desktop system," and I have turned up an interesting bit of info. (Note: My system is an AMD64 X2 2.5GHz running Linux.)

When I mount an XFS volume (thus loading the xfs kernel module), the kernel spawns two CPU-bound threads for "xfslogd" and "xfsdatad". However, it appears that only one of each of these kernel processes is getting any load, as indicated by "ps ax":

 3700 ?        S<     0:00 [xfslogd/0]
 3701 ?        S<     0:19 [xfslogd/1]
 3702 ?        S<     0:00 [xfsdatad/0]
 3703 ?        D<     0:05 [xfsdatad/1]

For each of these kernel threads, only those on CPU #2 are actually pulling notable load. Why is this?

I understand that I may have overlooked some critical tidbit of info in the man pages, or perhaps I have not yet found something online that could explain a default limitation. If so, I would very much enjoy new information about using XFS on a desktop system.

TYIA for your answer(s).

-- 
Mark

"What better place to find oneself than
 on the streets of one's home village?"
      --Capt. Jean-Luc Picard, "Family"

Re: XFS mkfs/mount options

[Dave Chinner]

On Tue, Jun 17, 2008 at 12:33:57AM -0700, Mark wrote:
> I have been doing some experiments with XFS on my "hot new desktop
> system," and I have turned up an interesting bit of info. (Note:
> My system is an AMD64 X2 2.5GHz running Linux.)
> 
> When I mount an XFS volume (thus loading the xfs kernel module),
> the kernel spawns two CPU-bound threads for "xfslogd" and
> "xfsdatad". However, it appears that only one of each of these
> kernel processes is getting any load, as indicated by "ps ax":
> 
>  3700 ?        S<     0:00 [xfslogd/0]
>  3701 ?        S<     0:19 [xfslogd/1]
>  3702 ?        S<     0:00 [xfsdatad/0]
>  3703 ?        D<     0:05 [xfsdatad/1]
> 
> For each of these kernel threads, only those on CPU #2 are actually pulling notable load. Why is this?

Because your disk interrupts are delivered to that CPU only.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS mkfs/mount options

[Mark]

--- On Tue, 6/17/08, Dave Chinner <david@fromorbit.com> wrote:

> > For each of these kernel threads, only those on CPU #2
> are actually pulling notable load. Why is this?
> 
> Because your disk interrupts are delivered to that CPU
> only.

Thank you! Tuning my IRQ delivery gave a noticeable speed-up.

-- 
Mark

"What better place to find oneself than
 on the streets of one's home village?"
      --Capt. Jean-Luc Picard, "Family"

Re: XFS mkfs/mount options

[Justin Piszcz]

How did you tune your IRQ delivery?

Justin.

On Tue, 17 Jun 2008, Mark wrote:

> --- On Tue, 6/17/08, Dave Chinner <david@fromorbit.com> wrote:
>
>>> For each of these kernel threads, only those on CPU #2
>> are actually pulling notable load. Why is this?
>>
>> Because your disk interrupts are delivered to that CPU
>> only.
>
> Thank you! Tuning my IRQ delivery gave a noticeable speed-up.
>
> -- 
> Mark
>
> "What better place to find oneself than
> on the streets of one's home village?"
>      --Capt. Jean-Luc Picard, "Family"
>
>

Re: XFS mkfs/mount options

[Mark]

--- On Tue, 6/17/08, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:

> From: Justin Piszcz <jpiszcz@lucidpixels.com>
> Subject: Re: XFS mkfs/mount options
> To: "Mark" <musicman529@yahoo.com>
> Cc: xfs@oss.sgi.com
> Date: Tuesday, June 17, 2008, 5:27 AM
> 
> How did you tune your IRQ delivery?

Generically:

echo X > /proc/irq/[IRQ#]/smp_affinity

Where X is a 32-bit hexadecimal bitmask.

My real procedure:

First, I confirmed that both drives (sda and sdb) were triggering different interrupts on the same CPU. "cat /dev/sda > /dev/null &" for activity, followed by "cat /proc/interrupts" a few times. (NOT AS ROOT!) Interrupt 20 was triggering only on the second CPU. Killed the background task.

Repeat, using "cat /dev/sdb > /dev/null &". Interrupt 21, also routing to the second CPU. Bottleneck likely confirmed.

A short hunt in /usr/src/linux/Documentation turned up the smp_affinity files. Looking at /proc/irq/2[01]/smp_affinity shows that both contain "ffffffff", that is, use all available CPU's.

To force the matter, I typed:

echo 00000001 > /proc/irq/21/smp_affinity
echo 00000002 > /proc/irq/20/smp_affinity

I dropped privileges, then repeated the "cat /dev..." above for both drives, confirming that interrupts were indeed going to CPU0 for int21 and CPU1 for int20.

Running a homebrewed multi-threading benchmark showed a possible speed-up for writes on XFS. I have not yet run "official" tests (Bonnie++ or my own) but will do so tonight. I expect the loss from cache-bouncing to be canceled out by the win from concurrent I/O.

-- 
Mark

"What better place to find oneself than
 on the streets of one's home village?"
      --Capt. Jean-Luc Picard, "Family"

Re: XFS mkfs/mount options

[Martin Steigerwald]

Hi Mark,

Am Dienstag 17 Juni 2008 schrieb Mark:
> --- On Tue, 6/17/08, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> > From: Justin Piszcz <jpiszcz@lucidpixels.com>
> > Subject: Re: XFS mkfs/mount options
> > To: "Mark" <musicman529@yahoo.com>
> > Cc: xfs@oss.sgi.com
> > Date: Tuesday, June 17, 2008, 5:27 AM
> >
> > How did you tune your IRQ delivery?
>
> Generically:
>
> echo X > /proc/irq/[IRQ#]/smp_affinity
>
> Where X is a 32-bit hexadecimal bitmask.
>
> My real procedure:

[... longer procedure ...]

We always use 

martin@shambala> apt-cache show irqbalance
Package: irqbalance
Priority: extra
Section: utils
Installed-Size: 64
Maintainer: Kyle McMartin <kyle@ubuntu.com>
Architecture: i386
Version: 0.55-2.1
Depends: debconf (>= 0.5) | debconf-2.0, libc6 (>= 2.7-1), libglib2.0-0 
(>= 2.12.0)
Filename: pool/main/i/irqbalance/irqbalance_0.55-2.1_i386.deb
Size: 17496
MD5sum: e4393d6ac2659e08f1bdecab2d94c1bf
SHA1: 4b49deb3b522dec88c820bde78cfb6abb20365dd
SHA256: 069104aff5561f09c8affcfe2aa9e144bed359a7c935c71c9968c78d1e8bf1b3
Description: Daemon to balance interrupts for SMP systems
 Daemon to balance interrupts across multiple CPUs, which can lead to
 better performance and IO balance on SMP systems. This package is
 especially useful on systems with multi-core processors, as interrupts
 will typically only be serviced by the first core.
Tag: admin::hardware, admin::kernel, interface::commandline, 
interface::daemon, network::server, role::program

for this kind of task or well the version in Debian Etch of it, the former 
is Lenny/Sid:

Version: 0.12-7
Depends: libc6 (>= 2.3.6-6), debconf (>= 0.5) | debconf-2.0

Is there any need to fine-tune interrupts manually?

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7