* Unaligned AIO/DIO
@ 2011-05-31 18:22 Troels Arvin
2011-06-01 10:46 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Troels Arvin @ 2011-05-31 18:22 UTC (permalink / raw)
To: fio
Hello,
Running fio 1.55 on RHEL 6.1 x86_64 with a multipath FC storage backend
on an EXT4 file system.
Some minutes after starting fio, I see the following in syslog:
kernel: EXT4-fs (dm-28): Unaligned AIO/DIO on inode 4063238 by fio;
performance will be poor.
Should I be worried? Is there something wrong with my storage layout?
Or am I using fio in strange way?
My job file:
[changed-from-iometer-file-access-server]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256
write_bw_log
write_lat_log
numjobs=6
--
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Unaligned AIO/DIO
2011-05-31 18:22 Unaligned AIO/DIO Troels Arvin
@ 2011-06-01 10:46 ` Jens Axboe
2011-06-01 12:51 ` Troels Arvin
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2011-06-01 10:46 UTC (permalink / raw)
To: Troels Arvin; +Cc: fio
On 2011-05-31 20:22, Troels Arvin wrote:
> Hello,
>
> Running fio 1.55 on RHEL 6.1 x86_64 with a multipath FC storage backend on an EXT4 file system.
>
> Some minutes after starting fio, I see the following in syslog:
> kernel: EXT4-fs (dm-28): Unaligned AIO/DIO on inode 4063238 by fio; performance will be poor.
>
> Should I be worried? Is there something wrong with my storage layout?
> Or am I using fio in strange way?
>
> My job file:
>
> [changed-from-iometer-file-access-server]
> bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
> rw=randrw
> rwmixread=70
> direct=1
> size=10g
> ioengine=libaio
> iodepth=256
> write_bw_log
> write_lat_log
> numjobs=6
Do you have the RHEL6.1 kernel sources handy? That message is not in the
upstream kernel. Unaligned could be memory, or it could be offset/size.
In general O_DIRECT does not work for unaligned IO. Perhaps it's
referring to writes that aren't stripe aligned, but that seems fairly
expected and something dm should handle.
So to render an opinion on this, I'd need to see a copy of the file that
prints that message :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Unaligned AIO/DIO
2011-06-01 10:46 ` Jens Axboe
@ 2011-06-01 12:51 ` Troels Arvin
2011-06-01 13:17 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Troels Arvin @ 2011-06-01 12:51 UTC (permalink / raw)
To: fio
Hello,
Jens Axboe wrote:
> Do you have the RHEL6.1 kernel sources handy?
I just grabbed the particular kernel package'es source-package,
installed it into ~/rpmbuild and ran "rpmbuild -bp" on its spec-file
which should result in all RHs patches to be applied.
From the resulting code in ~/rpmbuild/BUILD/..., I grabbed file.c and
placed it here:
http://troels.arvin.dk/misc/fio/unaligned/
Lines close to the error message (probably wrapped by my mailer):
/* Unaligned direct AIO must be serialized; see comment above */
if (unaligned_aio) {
static unsigned long unaligned_warn_time;
/* Warn about this once per day */
if (printk_timed_ratelimit(&unaligned_warn_time, 60*60*24*HZ))
ext4_msg(inode->i_sb, KERN_WARNING,
"Unaligned AIO/DIO on inode %ld by %s; "
"performance will be poor.",
inode->i_ino, current->comm);
mutex_lock(&EXT4_I(inode)->i_aio_mutex);
ext4_aiodio_wait(inode);
}
--
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Unaligned AIO/DIO
2011-06-01 12:51 ` Troels Arvin
@ 2011-06-01 13:17 ` Jens Axboe
2011-06-01 14:33 ` Troels Arvin
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2011-06-01 13:17 UTC (permalink / raw)
To: Troels Arvin; +Cc: fio
On 2011-06-01 14:51, Troels Arvin wrote:
> Hello,
>
> Jens Axboe wrote:
>> Do you have the RHEL6.1 kernel sources handy?
>
> I just grabbed the particular kernel package'es source-package, installed it into ~/rpmbuild and ran "rpmbuild -bp" on its spec-file which should result in all RHs patches to be applied.
>
> From the resulting code in ~/rpmbuild/BUILD/..., I grabbed file.c and placed it here:
> http://troels.arvin.dk/misc/fio/unaligned/
>
> Lines close to the error message (probably wrapped by my mailer):
>
> /* Unaligned direct AIO must be serialized; see comment above */
> if (unaligned_aio) {
> static unsigned long unaligned_warn_time;
>
> /* Warn about this once per day */
> if (printk_timed_ratelimit(&unaligned_warn_time, 60*60*24*HZ))
> ext4_msg(inode->i_sb, KERN_WARNING,
> "Unaligned AIO/DIO on inode %ld by %s; "
> "performance will be poor.",
> inode->i_ino, current->comm);
>
> mutex_lock(&EXT4_I(inode)->i_aio_mutex);
> ext4_aiodio_wait(inode);
> }
Ah, so this is indeed the < 4KB O_DIRECT writes. The iometer like test
case you are using is meant to be used on a raw block device, not on a
file system. For a file system, doing less than the block size of IO
will incur RMW cycles and thus be very slow compared to full block size
(or more) O_DIRECT writes.
If you want to run it on the fs, either change the block size mappings
to avoid anything less than 4K. Or change the fs to be formatted with
512b blocks.
Or just run the test on the raw device :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Unaligned AIO/DIO
2011-06-01 13:17 ` Jens Axboe
@ 2011-06-01 14:33 ` Troels Arvin
2011-06-02 11:51 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Troels Arvin @ 2011-06-01 14:33 UTC (permalink / raw)
To: fio
Hello,
Jens Axboe wrote:
> The iometer like test
> case you are using is meant to be used on a raw block device, not on a
> file system.
OK, but how does one use fio with a raw device? Can you give an example
of a job specification file for working with a raw block device?
And can that be done in a way which simulates parallel I/O (several
processes/threads working simultaneously)?
--
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Unaligned AIO/DIO
2011-06-01 14:33 ` Troels Arvin
@ 2011-06-02 11:51 ` Jens Axboe
0 siblings, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2011-06-02 11:51 UTC (permalink / raw)
To: Troels Arvin; +Cc: fio
On 2011-06-01 16:33, Troels Arvin wrote:
> Hello,
>
> Jens Axboe wrote:
>> The iometer like test
>> case you are using is meant to be used on a raw block device, not on a
>> file system.
>
> OK, but how does one use fio with a raw device? Can you give an example of a job specification file for working with a raw block device?
To take your example:
[global]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256
write_bw_log
write_lat_log
numjobs=6
[sda]
filename=/dev/sda
would run the workload in question directly on the raw sda device.
> And can that be done in a way which simulates parallel I/O (several
> processes/threads working simultaneously)?
For the above example, you have 6 processes doing random read/write
patterns with the 70/30 read/write mix and with the block sizes given.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-06-02 11:51 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-05-31 18:22 Unaligned AIO/DIO Troels Arvin
2011-06-01 10:46 ` Jens Axboe
2011-06-01 12:51 ` Troels Arvin
2011-06-01 13:17 ` Jens Axboe
2011-06-01 14:33 ` Troels Arvin
2011-06-02 11:51 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox