new ext4 filesystem vs. converted ext3

new ext4 filesystem vs. converted ext3

[Micah Anderson]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]


Hello,

I recently ran into the subdirectory scalability issue of ext3 on our
listserver. Thankfully, ext4 exists, so it was finally time to move to
it. I followed the published process to convert the ext3 filesystem to
ext4[0] and things worked well, I'm no longer hitting the 32000 limit on
subdirectories.

However, ever since I did that change, I've noticed an increase in I/O
wait state on the CPUs. I've been trying to determine why, and if there
were some things I should tune on this ext4 filesystem.

First thing I found was that the Filesystem Features of an ext4
filesystem that were converted from ext3 are different than those that
are set on a filesystem that was created as ext4 (on Debian Squeeze),
From tune2fs -l:

a newly created ext4 has:
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super huge_file  uninit_bg dir_nlink extra_isize

my migrated one has: 
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file uninit_bg

The difference being new filesystems have 'flex_bg', 'huge_file' and
'extra_isize' set, and the converted filesystem has 'large_file' and
'uninit_bg'. I'm not sure I understand why the difference and wonder if
someone can explain to me why? Or if I should be changing these?

Currently the filesystem is mounted with just rw,relatime set, and I
wonder if there is a combination of some tune2fs changes, and mount
options that we would benefit from.

Basically this filesystem has a lot of small files, there is a set of
spool directories where there are a lot of small writes, small reads and
deletes. There is also an archive of list postings, which is a fairly
large set of small files, which is mostly small writes, reads every once
and a while. There is also a series of stats that happen on a large
number of subdirectories.

Thank you for any suggestions!
micah


0. https://ext4.wiki.kernel.org/index.php/Ext4_Howto#Converting_an_ext3_filesystem_to_ext4

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: new ext4 filesystem vs. converted ext3

[Ted Ts'o]

On Fri, Jun 03, 2011 at 05:49:17PM -0400, Micah Anderson wrote:
> However, ever since I did that change, I've noticed an increase in I/O
> wait state on the CPUs. I've been trying to determine why, and if there
> were some things I should tune on this ext4 filesystem.

How are you measuring this?

						- Ted

Re: new ext4 filesystem vs. converted ext3

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

On Fri, 3 Jun 2011 23:09:49 -0400, Ted Ts'o <tytso@mit.edu> wrote:
> On Fri, Jun 03, 2011 at 05:49:17PM -0400, Micah Anderson wrote:
> > However, ever since I did that change, I've noticed an increase in I/O
> > wait state on the CPUs. I've been trying to determine why, and if there
> > were some things I should tune on this ext4 filesystem.
> 
> How are you measuring this?

Through munin graphs, unfortunately we don't have a lot of data from
before the change, but you can see the jump early on in this graph,
where the change was made:

http://lackof.org/~taggart/tmp/willet-cpu-year.png

I'll note that we also moved to squeeze from lenny at this
time. Basically we decided to move to squeeze and then convert to ext4,
so that throws in some other variables here too.

As I mentioned before, this is a high traffic mailing list system, which
does a lot of I/O. We're also seeing lots of rescheduling interrupts
after the upgrade to the squeeze kernel:

http://lackof.org/~taggart/tmp/willet-irqstats-year.png

(see the jump in the pink line on the left side?)

Searches on "rescheduling interrupts" finds lots of people using
powertop to try to reduce their laptop power usage (this might be
related: https://help.ubuntu.com/community/ReschedulingInterrupts), I'm
wondering if maybe there were changes from lenny to squeeze to try and
optimize for that, but are hurting in our case.

micah



[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: new ext4 filesystem vs. converted ext3

[Ted Ts'o]

On Mon, Jun 06, 2011 at 12:43:10PM -0400, micah anderson wrote:
> 
> Through munin graphs, unfortunately we don't have a lot of data from
> before the change, but you can see the jump early on in this graph,
> where the change was made:
> 
> http://lackof.org/~taggart/tmp/willet-cpu-year.png
> 
> I'll note that we also moved to squeeze from lenny at this
> time. Basically we decided to move to squeeze and then convert to ext4,
> so that throws in some other variables here too.
> 
> As I mentioned before, this is a high traffic mailing list system, which
> does a lot of I/O. We're also seeing lots of rescheduling interrupts
> after the upgrade to the squeeze kernel:
> 
> http://lackof.org/~taggart/tmp/willet-irqstats-year.png

Oh, I bet I know what's going on.  Ext3 defaults to barriers being
off.  Ext4 defaults to barriers turned on, which is safer if you have
power drops.  If you have a UPS and are confident that the UPS
monitoring software is properly setup so the system will go through a
controlled, clean shutdown when the UPS power is running low, then you
could consider disabling barriers on ext4 without committing
professional sysadmin malpractice.  :-)

Since mail systems tend to be very fsync() happy, and fsyncs()
translate to barriers, that's probably the explanation of what's going
on here.

							- Ted