Help creating filesystem (xfs) and partitioning

Help creating filesystem (xfs) and partitioning

[Roberto Spadim]

Hi guys i have a server with two 500gb hdd (enterprise sata 7200rpm)
It's a dell R210 II Server, intel e3 1220v2 processor
There's no raid card just mainboard, disk, ram memory (8gb) and four
network interfaces (internet and bond of local network)
I will use it as a server (mariadb,php,apache) with slackware linux
and kernel 3.9.10, mariadb with aria storage.
I think kernel 3.10.2 isn't nice yet to work with raid1, it have some
bugs, right?

Well the system setup...

I want 4 partitions in each disk
/boot with 200mb, xfs filesystem and raid1
swap with 20gb (or 10gb),
/ with 100gb, xfs filesystem and raid1
/home with all disk left (near to 400gb), xfs filesystem and raid1

No lvm, i will not expand filesystem, just replace disk when it crash
(maybe next 4 to 5 years), i will not expand hardware, just replace
when crashed too (network card, memory, cpu, etc)
After 5 year i think that will have ~100GB in /home or less

I tested this system in raid1, raid10, raid5 and raid6, with last
archlinux, the best performace was raid1
There's many non sequencial writes, with many thread reading/writing
to disk, that's the main reason that i think raid1 worked better

Now the point is, what's the best fdisk, mdadm, mkfs_xfs, tunefs,
smartctl and others tune tools and /sys /proc parameters?
I want a good align and performace

What more information should i share?
Thanks guys

--
Roberto Spadim

Re: Help creating filesystem (xfs) and partitioning

[Stan Hoeppner]

On 7/23/2013 7:06 PM, Roberto Spadim wrote:

> /boot with 200mb, xfs filesystem and raid1

Using a journaling filesystem for /boot is pointless.  Use EXT2 instead.

> swap with 20gb (or 10gb),
> / with 100gb, xfs filesystem and raid1
> /home with all disk left (near to 400gb), xfs filesystem and raid1

It is ill advised to put more than one XFS filesystem on a single
(effective) spindle which is what RAID1 is.  Doing so results in too
many allocation groups, which results in head thrashing, which results
in decreased performance.  Use a single XFS for rootfs and home, manage
space by enabling and setting quotas.

...
> Now the point is, what's the best fdisk, mdadm, mkfs_xfs, tunefs,
> smartctl and others tune tools and /sys /proc parameters?
> I want a good align and performace

None of these will increase your performance with a RAID1 pair.
Switching your elevator from CFQ to deadline, and making sure NCQ is
enabled and working properly, is about the best you can do to maximize
md RAID1 performance.  And using a single XFS as I described above,
instead of 2 or 3.

If your workload(s) involve heavy mixed metadata and data changes, i.e.
untar a kernel tarball then delete the directory tree, then mounting
with inode64 will improve performance a bit.  inode64 has been made the
default XFS allocator but I'm not sure with which kernel version that
took affect.  To find out simply do "cat /proc/mounts".

See mkfs.xfs(8), mount(8)
http://xfs.org/index.php/XFS_FAQ
http://xfs.org/docs/xfsdocs-xml-dev/XFS_Labs/tmp/en-US/html/xfs-quotas-lab.html

-- 
Stan

Re: Help creating filesystem (xfs) and partitioning

[Stan Hoeppner]

On 7/24/2013 11:06 AM, Roberto Spadim wrote:
> What about fdisk align?

If the disks have 512 bytes per physical sector, the standard for over
30 years, then there is nothing to align.  If the disks have 4096 bytes
per physical sector and 512 per logical sector, i.e. Advanced Format
disks, then you need to make sure your partitions are aligned to 4K
boundaries.  Start your first partition at LBA sector 2048 and you
should be golden.  And make sure each partition is evenly divisible by
4096 bytes.  If you don't partition AF disks properly your 4K filesystem
blocks will straddle two adjacent physical 4K sectors and cause drive
internal read-modify-write.  This can cut performance by up to 50%.

> Em 24/07/2013 03:56, "Stan Hoeppner" <stan@hardwarefreak.com> escreveu:
>>
>> On 7/23/2013 7:06 PM, Roberto Spadim wrote:
>>
>>> /boot with 200mb, xfs filesystem and raid1
>>
>> Using a journaling filesystem for /boot is pointless.  Use EXT2 instead.
>>
>>> swap with 20gb (or 10gb),
>>> / with 100gb, xfs filesystem and raid1
>>> /home with all disk left (near to 400gb), xfs filesystem and raid1
>>
>> It is ill advised to put more than one XFS filesystem on a single
>> (effective) spindle which is what RAID1 is.  Doing so results in too
>> many allocation groups, which results in head thrashing, which results
>> in decreased performance.  Use a single XFS for rootfs and home, manage
>> space by enabling and setting quotas.
>>
>> ...
>>> Now the point is, what's the best fdisk, mdadm, mkfs_xfs, tunefs,
>>> smartctl and others tune tools and /sys /proc parameters?
>>> I want a good align and performace
>>
>> None of these will increase your performance with a RAID1 pair.
>> Switching your elevator from CFQ to deadline, and making sure NCQ is
>> enabled and working properly, is about the best you can do to maximize
>> md RAID1 performance.  And using a single XFS as I described above,
>> instead of 2 or 3.
>>
>> If your workload(s) involve heavy mixed metadata and data changes, i.e.
>> untar a kernel tarball then delete the directory tree, then mounting
>> with inode64 will improve performance a bit.  inode64 has been made the
>> default XFS allocator but I'm not sure with which kernel version that
>> took affect.  To find out simply do "cat /proc/mounts".
>>
>> See mkfs.xfs(8), mount(8)
>> http://xfs.org/index.php/XFS_FAQ
>>
> http://xfs.org/docs/xfsdocs-xml-dev/XFS_Labs/tmp/en-US/html/xfs-quotas-lab.html
>>
>> --
>> Stan
>>
>>
> 

Re: Help creating filesystem (xfs) and partitioning

[Stan Hoeppner]

On 7/24/2013 12:07 PM, Roberto Spadim wrote:

> Could i use ext3 or ext4 for boot filesystem?  I need crash security (ok a
> ups is the best solution) xfs could be used too?

Again, Roberto, a journaling filesystem is not necessary for /boot.  It
will not make files in /boot any more crash resistant.  Files in /boot
are only modified when you replace (upgrade) your kernel.  Corruption
can only occur when file writes are in flight but not completely on
disk.  Therefore, to corrupt files in /boot, the machine must crash or
lose power while you're installing a new kernel, i.e. within a window of
about 10 seconds.  There are 31,536,000 seconds in a year.  So the odds
of a crash/power loss during any one of your kernel upgrades are, if my
math is close, approximately

1 in 3,153,600

> If i need a home directory with crypt filesystem, what should i use? Two
> partitions one with xfs and other with dmcrypt (for example) and xfs, or
> only one partition and a loop mount with crypt and quota limit for home
> directory?

Is this a workstation, personal server, or multiuser server with decent
load?  If the load is light you can get away with multiple XFS
filesystem on a single spindle, separate rootfs and home.

-- 
Stan

Re: Help creating filesystem (xfs) and partitioning

[Mark Knecht]

On Wed, Jul 24, 2013 at 12:35 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 7/24/2013 12:07 PM, Roberto Spadim wrote:
>
>> Could i use ext3 or ext4 for boot filesystem?  I need crash security (ok a
>> ups is the best solution) xfs could be used too?
>
> Again, Roberto, a journaling filesystem is not necessary for /boot.  It
> will not make files in /boot any more crash resistant.  Files in /boot
> are only modified when you replace (upgrade) your kernel.  Corruption
> can only occur when file writes are in flight but not completely on
> disk.  Therefore, to corrupt files in /boot, the machine must crash or
> lose power while you're installing a new kernel, i.e. within a window of
> about 10 seconds.  There are 31,536,000 seconds in a year.  So the odds
> of a crash/power loss during any one of your kernel upgrades are, if my
> math is close, approximately
>
> 1 in 3,153,600
>

+1

Also, at least on my personal systems, nothing resides on /boot that
doesn't reside somewhere else under /. The kernels & associated files
are in /usr/src. I periodically will copy grub.conf to /root for safe
keeping. Personally I don't even back up /boot anymore as the biggest
issue I always have if I lose a disk or make some horrible fat
fingered mistake is to reinstall grub which (I believe to be true)
won't be effected by the filesystem choice.

Just my POV.

Cheers,
Mark

Re: Help creating filesystem (xfs) and partitioning

[Roberto Spadim]

sorry i was sending as html (i was at android gmail), now as text...

-------
Nice other doubts

What about fdisk align?
Could i use ext3 or ext4 for boot filesystem? I need crash security
(ok a ups is the best solution) xfs could be used too?
Or maybe.. Could i use only two partitions, swap at beginning of
disks, and the raid1 with xfs partition?

If i need a home directory with crypt filesystem, what should i use?
Two partitions one with xfs and other with dmcrypt (for example) and
xfs, or only one partition and a loop mount with crypt and quota limit
for home directory?

Thanks guys

Re: Help creating filesystem (xfs) and partitioning

[Bill Davidsen]

Roberto Spadim wrote:
> Hi guys i have a server with two 500gb hdd (enterprise sata 7200rpm)
> It's a dell R210 II Server, intel e3 1220v2 processor
> There's no raid card just mainboard, disk, ram memory (8gb) and four
> network interfaces (internet and bond of local network)
> I will use it as a server (mariadb,php,apache) with slackware linux
> and kernel 3.9.10, mariadb with aria storage.
> I think kernel 3.10.2 isn't nice yet to work with raid1, it have some
> bugs, right?
>
I don't think this is a bad choice, although the 3.10 may be a bit faster for 
many CPU bound processes, even if only for ms at a time. If you like it better 
it's a good choice.

> Well the system setup...
>
> I want 4 partitions in each disk
> /boot with 200mb, xfs filesystem and raid1
> swap with 20gb (or 10gb),
> / with 100gb, xfs filesystem and raid1
> /home with all disk left (near to 400gb), xfs filesystem and raid1
>
Other people have mentioned not using two filesystems to prevent thrashing and 
head motion, The only thing in root which changes a lot are logs, and then only 
when comthing log generating is running. Certainly http, nntp, and mail servers 
qualify. Unless you see some reason I missed, joining all the root and home in 
one filesystem will probably be a win over having two.

> No lvm, i will not expand filesystem, just replace disk when it crash
> (maybe next 4 to 5 years), i will not expand hardware, just replace
> when crashed too (network card, memory, cpu, etc)
> After 5 year i think that will have ~100GB in /home or less
>
> I tested this system in raid1, raid10, raid5 and raid6, with last
> archlinux, the best performace was raid1

I note the lack of raid10, particularly raid10 with -f2 (far two) layout. In 
general that is going to be faster than raid1, and equally robust. You ran a lot 
of benchmarks, one more would config your choice or give you a better option.

> There's many non sequencial writes, with many thread reading/writing
> to disk, that's the main reason that i think raid1 worked better
>
> Now the point is, what's the best fdisk, mdadm, mkfs_xfs, tunefs,
> smartctl and others tune tools and /sys /proc parameters?
> I want a good align and performace
>
Then you can play with hardware and software readahead, stride, all the 
write{back,thru,etc} options xfs may provide. Frankly, if you want good disk 
performance, the most time effective thing you can do is add more drives to 
spread the head motion and do more parallel i/o. With only two drives and raid1 
you are limited to the transfer rate of the slowest drive and sequential seeks. 
with more spindles you start to do things in parallel, use the outer part of the 
drive for faster transfer, etc. What you have has a low theoretical max which 
you can beat without tuning using more disk and raid10.

> What more information should i share?

Write size. The will effect your chunk size choices. I don't know what xfs gives 
you in control of favoring writes of a given size, ext4 does let you provide 
information to the kernel in extended options, which can be useful. In my 
experience more so when doing a series of sequential small writes (more than a 
chunk total, not huge) than a ton of small scattered writes, just mentioning 
things you can evaluate, since you have put a lot of work into test already.

> Thanks guys
>
> --
> Roberto Spadim
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Help creating filesystem (xfs) and partitioning

[Roberto Spadim]

> I don't think this is a bad choice, although the 3.10 may be a bit faster
> for many CPU bound processes, even if only for ms at a time. If you like it
> better it's a good choice.
i will use xeon e3 1220v2 (without hyper thread), there's a 'dummy'
list of cpu/schedulers improvements?


> Other people have mentioned not using two filesystems to prevent thrashing
> and head motion, The only thing in root which changes a lot are logs, and
> then only when comthing log generating is running. Certainly http, nntp, and
> mail servers qualify. Unless you see some reason I missed, joining all the
> root and home in one filesystem will probably be a win over having two.
ok i think quotas is nice, what about changing /boot / and /home, to
only one partition? instead of 3 partitions?


> I note the lack of raid10, particularly raid10 with -f2 (far two) layout. In
> general that is going to be faster than raid1, and equally robust. You ran a
> lot of benchmarks, one more would config your choice or give you a better
> option.
hum i tested the system in raid10 and it runs slower with far and near
layout, i think that it have many read in non consecutive blocks ,
that make far layout bad
raid1 runs nice without problem and read aren't fast but they are
intensive (many files reads instead of many information in one file)


> Then you can play with hardware and software readahead, stride, all the
> write{back,thru,etc} options xfs may provide. Frankly, if you want good disk
> performance, the most time effective thing you can do is add more drives to
> spread the head motion and do more parallel i/o. With only two drives and
> raid1 you are limited to the transfer rate of the slowest drive and
> sequential seeks. with more spindles you start to do things in parallel, use
> the outer part of the drive for faster transfer, etc. What you have has a
> low theoretical max which you can beat without tuning using more disk and
> raid10.
no problem, a single disk works for the system, raid1 is for security,
maybe a bcache or another ssd+hdd cache could allow a better
performace, but i don't think i will need it
i don't know what's the best xfs parameters / fdisk parameters to
align disk with filesystem, should i add some parameter?


>> What more information should i share?
> Write size. The will effect your chunk size choices. I don't know what xfs
> gives you in control of favoring writes of a given size, ext4 does let you
> provide information to the kernel in extended options, which can be useful.
> In my experience more so when doing a series of sequential small writes
> (more than a chunk total, not huge) than a ton of small scattered writes,
> just mentioning things you can evaluate, since you have put a lot of work
> into test already.
well write mean size is something near to 4kb or 1kb, with some big
writes (1mb) when using a big commit or something like it in innodb
tables


thanks Bill!
any other idea is wellcome too =)

-- 
Roberto Spadim

Re: Help creating filesystem (xfs) and partitioning

[Stan Hoeppner]

On 7/25/2013 5:36 PM, Roberto Spadim wrote:

> i don't know what's the best xfs parameters / fdisk parameters to
> align disk with filesystem, should i add some parameter?

You don't align a filesystem with fdisk.  fdisk creates partitions.  As
I said earlier, start your first partition at sector 2048, and create
each successive partition to be of a size evenly divisible by 4096 bytes.

WRT mkfs.xfs, use the defaults.  It reads the md config and configures
automatically.  You only need to specify command line values when using
hardware RAID or LVM atop hard or software RAID.

And in your case, with only two mirrored spindles, you simply can't do
anything to tweak XFS performance anyway.

-- 
Stan

Re: Help creating filesystem (xfs) and partitioning

[Roberto Spadim]

ok =] in other words no problem about performace loss, just use one
xfs per disk, right?
the second doubt... i was thinking about /boot /home and /, in the
same partition, should i consider other design? a /boot with ext2 or
ext3 and a / with xfs?

i'm considering only one partition with swap and one partition with
raid1+xfs, what you think?
in a crash (power failure) i will have better rescure with only one
partition? or many partitions?

Re: Help creating filesystem (xfs) and partitioning

[Stan Hoeppner]

On 7/25/2013 7:36 PM, Roberto Spadim wrote:
> ok =] in other words no problem about performace loss, just use one
> xfs per disk, right?
> the second doubt... i was thinking about /boot /home and /, in the
> same partition, should i consider other design? a /boot with ext2 or
> ext3 and a / with xfs?

What I would do:

	swap	1GB partition on each disk

md RAID1

/boot	ext2	100MB
rootfs	XFS	remainder of disk
home
...

> i'm considering only one partition with swap and one partition with
> raid1+xfs, what you think?

I wouldn't do it.  See above.

> in a crash (power failure) i will have better rescure with only one
> partition? or many partitions?

Having a separate /boot is always a good idea.

And truthfully, for a system of this caliber, you don't really gain
anything by using XFS, certainly not from a performance standpoint.  If
you were already an XFS user on large systems and it was simply your "go
to" filesystem, then using it on this system may make sense.  And if you
don't have a working UPS, you should definitely stay away from XFS.
Power failure shouldn't cause filesystem corruption, but it may well
corrupt or zero out files that are open for write but not written.  XFS
journals metadata, not data.

-- 
Stan

Re: Help creating filesystem (xfs) and partitioning

[Roberto Spadim]

> And truthfully, for a system of this caliber, you don't really gain
> anything by using XFS, certainly not from a performance standpoint.  If
> you were already an XFS user on large systems and it was simply your "go
> to" filesystem, then using it on this system may make sense.  And if you
> don't have a working UPS, you should definitely stay away from XFS.
> Power failure shouldn't cause filesystem corruption, but it may well
> corrupt or zero out files that are open for write but not written.  XFS
> journals metadata, not data.

i'm using xfs because i tested with ext4, xfs and reiserfs (v3) and
xfs was the fastest
i use UPS, but... well you know... some one can remove the power
cable... users sometime make mistakes :) hehe
i didn't tested the btrfs yet, i think it's not mature for production use

well power failure is a problem in any filesystem... what filesystem
you consider is the "best"?

considering that:
i'm running filesystem over md raid1
best = good power failure
files with >40gb (some mysql tables are big)
a big directory structure (root directory with man pages libs, linux
kernel and some packages that i compile (php, mariadb, apache), and
others linux tools, etc...)
a home directory with maybe many temporary files, mysql sometime
create temporary files for query sorting, in this case crud operations
happens very often, create file/put data/read/delete file

well i think it's a test scenario, but experiences/ideas are wellcome

-- 
Roberto Spadim

Re: Help creating filesystem (xfs) and partitioning

[Stan Hoeppner]

On 7/25/2013 8:47 PM, Roberto Spadim wrote:
>> And truthfully, for a system of this caliber, you don't really gain
>> anything by using XFS, certainly not from a performance standpoint.  If
>> you were already an XFS user on large systems and it was simply your "go
>> to" filesystem, then using it on this system may make sense.  And if you
>> don't have a working UPS, you should definitely stay away from XFS.
>> Power failure shouldn't cause filesystem corruption, but it may well
>> corrupt or zero out files that are open for write but not written.  XFS
>> journals metadata, not data.
> 
> i'm using xfs because i tested with ext4, xfs and reiserfs (v3) and
> xfs was the fastest
> i use UPS, but... well you know... some one can remove the power
> cable... users sometime make mistakes :) hehe
> i didn't tested the btrfs yet, i think it's not mature for production use
> 
> well power failure is a problem in any filesystem... what filesystem
> you consider is the "best"?

This depends on your use case and workloads.  EXT4 isn't suitable for
large multi-TB filesystems with highly parallel workloads whereas XFS
excels with these.  Neither XFS nor EXT4 are suitable for USB thumb drives.

> considering that:
> i'm running filesystem over md raid1
> best = good power failure
> files with >40gb (some mysql tables are big)
> a big directory structure (root directory with man pages libs, linux
> kernel and some packages that i compile (php, mariadb, apache), and
> others linux tools, etc...)
> a home directory with maybe many temporary files, mysql sometime
> create temporary files for query sorting, in this case crud operations
> happens very often, create file/put data/read/delete file
> 
> well i think it's a test scenario, but experiences/ideas are wellcome

As long as you have a good working UPS and barriers enabled (the
default) and working, then XFS should be fine.  XFS does a barrier write
test during mount.  If the test fails it will log an error in dmesg and
disable barriers.  This obviously requires corrective action.  Running
without barriers can potentially cause serious metadata corruption and
wreck the filesystem beyond repair due to drive caches not being
flushed.  If the test passes, barriers are enabled, nothing is logged,
and you're good to go.

-- 
Stan

Re: Help creating filesystem (xfs) and partitioning

[Roberto Spadim]

ok, well, i think i will continue the old style... xfs+raid1
the only point is

only one partition for /
two partitions for /boot /
three partitions for /boot /home /

Re: Help creating filesystem (xfs) and partitioning

[Bill Davidsen]

Roberto Spadim wrote:
> ok =] in other words no problem about performace loss, just use one
> xfs per disk, right?
> the second doubt... i was thinking about /boot /home and /, in the
> same partition, should i consider other design? a /boot with ext2 or
> ext3 and a / with xfs?
>
> i'm considering only one partition with swap and one partition with
> raid1+xfs, what you think?
> in a crash (power failure) i will have better rescure with only one
> partition? or many partitions?
>
I would create a /boot partition, root partition, and swap partition on each 
drive and make rais1 to create three devices, with two filesystem, /boot and 
root. My thought is that you rarely change /boot, and that MUST work to get the 
system up, so put it by itself. Also, I'm pretty sure you want ext on the boot 
filesystem, if for no other reason than the code in grub being better tested 
booting from that than any other. I will let someone who is more expert in xfs 
than I discuss any questions you have, boot issues may be urban myth, but I see 
no benefit to using anything else.

Note that I have no solid evidence the other than ext for /boot causes problems, 
but years of trouble free with ext3.

-- 
Bill Davidsen <davidsen@tmr.com>
   "'Nothing to hide' does not imply 'nothing to fear'"
       - me
   "AT&T could not seriously contend that a reasonable entity in its position
    could have believed that the alleged domestic dragnet was legal."
       -judge Vaughn R. Walker of the U.S. District Court
        for the Northern District of California, EFF vs. AT&T