GIT PULL] ext4 updates for 3.4

GIT PULL] ext4 updates for 3.4

[Theodore Ts'o]

The following changes since commit 62aa2b537c6f5957afd98e29f96897419ed5ebab:

  Linux 3.3-rc2 (2012-01-31 13:31:54 -0800)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git ext4_for_linus

for you to fetch changes up to 9d547c35799a4ddd235f1565cec2fff6c9263504:

  vfs: remove unused superblock helpers (2012-03-21 22:34:55 -0400)

----------------------------------------------------------------
Ext4 commits for 3.3 merge window; mostly cleanups and bug fixes

The changes to export dirty_writeback_interval are from Artem's s_dirt
cleanup patch series.  The same is true of the change to remove the
s_dirt helper functions which never got used by anyone in-tree.  I've
run these changes by Al Viro, and am carrying them so that Artem can
more easily fix up the rest of the file systems during the next merge
window.  (Originally we had hopped to remove the use of s_dirt from
ext4 during this merge window, but his patches had some bugs, so I
ultimately ended dropping them from the ext4 tree.)

----------------------------------------------------------------
Allison Henderson (1):
      ext4: correct ext4_punch_hole return codes

Artem Bityutskiy (5):
      ext4: do not mark superblock as dirty unnecessarily
      ext4: write superblock only once on unmount
      ext4: remove useless s_dirt assignment
      mm: export dirty_writeback_interval
      vfs: remove unused superblock helpers

Bobi Jam (1):
      ext4: expand commit callback and

Curt Wohlgemuth (3):
      ext4: mark possibly unused variable in ext4_mb_normalize_request()
      ext4: don't release page refs in ext4_end_bio()
      ext4: add comments to definition of ext4_io_end_t

Dan Carpenter (2):
      ext4: remove an unneeded NULL check in __ext4_check_dir_entry()
      ext4: using PTR_ERR() on the wrong variable in ext4_ext_migrate()

Darrick J. Wong (1):
      ext4: update s_free_{inodes,blocks}_count during online resize

Eric Sandeen (4):
      jbd2: clear BH_Delay & BH_Unwritten in journal_unmap_buffer
      ext4: remove the journal=update mount option
      ext4: remove the resize mount option
      ext4: avoid deadlock on sync-mounted FS w/o journal

Heiko Carstens (1):
      ext4: fix balloc.c printk-format-warning

Jan Kara (8):
      jbd2: split updating of journal superblock and marking journal empty
      jbd2: protect all log tail updates with j_checkpoint_mutex
      jbd2: issue cache flush after checkpointing even with internal journal
      jbd2: fix BH_JWrite setting in checkpointing code
      jbd2: declare __jbd2_journal_temp_unlink_buffer() static
      jbd2: remove always true condition in __journal_try_to_free_buffer()
      jbd2: remove bh_state lock from checkpointing code
      jbd2: cleanup journal tail after transaction commit

Jeff Moyer (3):
      ext4: fix race between unwritten extent conversion and truncate
      ext4: clean up the flags passed to __blockdev_direct_IO
      ext4: fix race between sync and completed io work

Joe Perches (3):
      ext4: remove redundant "EXT4-fs: " from uses of ext4_msg
      ext4: add no_printk argument validation, fix fallout
      ext4: avoid output message interleaving in ext4_error_<foo>()

Lukas Czerner (9):
      ext4: ignore EXT4_INODE_JOURNAL_DATA flag with delalloc
      ext4: fix resize when resizing within single group
      ext4: rewrite punch hole to use ext4_ext_remove_space()
      ext4: remove unused code from ext4_ext_map_blocks()
      ext4: give more helpful error message in ext4_ext_rm_leaf()
      ext4: fix start and len arguments handling in ext4_trim_fs()
      ext4: fix trimmed block count accunting
      ext4: always set then trimmed blocks count into len
      ext4: remove restrictive checks for EOFBLOCKS_FL

Santosh Nayak (1):
      ext4: Fix endianness bug when reading the MMP block

Seiji Aguchi (1):
      jbd2: add drop_transaction/update_superblock_end tracepoints

Theodore Ts'o (18):
      ext4: fold ext4_claim_inode into ext4_new_inode
      ext4: fix race when setting bitmap_uptodate flag
      ext4: fix INCOMPAT feature codepoint reservation for INLINEDATA
      ext4: remove EXT4_MB_{BITMAP,BUDDY} macros
      ext4: remove deprecation warnings for minix_df and grpid
      ext4: remove Opt_ignore
      ext4: remove the I_VERSION mount flag and use the super_block flag instead
      ext4: simplify handling of the errors=* mount options
      ext4: unify handling of mount options which have been removed
      ext4: use a table-driven handler for mount options
      ext4: move ext4_show_options() after parse_options()
      ext4: make ext4_show_options() be table-driven
      ext4: add debugging /proc file showing file system options
      ext4: ignore mount options supported by ext2/3 (but have since been removed)
      ext4: try to deprecate noacl and noxattr_user mount options
      ext4: check for zero length extent
      ext4: remove trailing newlines from ext4_msg() and ext4_error() messages
      ext4: change some printk() calls to use ext4_msg() instead

Yongqiang Yang (3):
      jbd2: allocate transaction from separate slab cache
      jbd2: rename functions which initialize slab caches
      jbd2: use KMEM_CACHE instead of kmem_cache_create()

Zheng Liu (2):
      ext4: remove unneeded variable in ext4_xattr_check_block()
      ext4: format flag in dx_probe()

 Documentation/filesystems/ext4.txt |    8 -
 fs/ext4/balloc.c                   |   63 ++-
 fs/ext4/dir.c                      |   13 +-
 fs/ext4/ext4.h                     |   34 +-
 fs/ext4/ext4_extents.h             |    4 +-
 fs/ext4/ext4_jbd2.h                |  128 ++++-
 fs/ext4/extents.c                  |  330 +++++-------
 fs/ext4/fsync.c                    |    2 +
 fs/ext4/ialloc.c                   |  260 ++++-----
 fs/ext4/inode.c                    |   95 ++--
 fs/ext4/mballoc.c                  |  342 +++++-------
 fs/ext4/mballoc.h                  |   20 +-
 fs/ext4/migrate.c                  |    2 +-
 fs/ext4/mmp.c                      |    4 +-
 fs/ext4/namei.c                    |    2 +-
 fs/ext4/page-io.c                  |   18 +-
 fs/ext4/resize.c                   |   37 +-
 fs/ext4/super.c                    | 1075 ++++++++++++++++--------------------
 fs/ext4/xattr.c                    |   25 +-
 fs/jbd2/checkpoint.c               |  140 +----
 fs/jbd2/commit.c                   |   47 ++-
 fs/jbd2/journal.c                  |  361 ++++++++-----
 fs/jbd2/recovery.c                 |    5 +-
 fs/jbd2/revoke.c                   |   12 +-
 fs/jbd2/transaction.c              |   48 ++-
 include/linux/fs.h                 |   13 -
 include/linux/jbd2.h               |   12 +-
 include/linux/journal-head.h       |    2 +
 include/trace/events/jbd2.h        |   29 +-
 mm/page-writeback.c                |    2 +
 30 files changed, 1522 insertions(+), 1611 deletions(-)

Re: GIT PULL] ext4 updates for 3.4

[Linus Torvalds]

On Mon, Mar 26, 2012 at 11:07 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>
> Ext4 commits for 3.3 merge window; mostly cleanups and bug fixes

Ted, is there any chance that the dirty handling (or something else)
has some serious latency bug?

I'm compiling an allmodconfig kernel, and my mouse is (sometimes) very
erratic with bad latency. And the biggest changes in the kernel I'm
running on that machine now (HEAD=529b73fc0a97) and the previous boot
of that machine (HEAD=e22057c85993) are the ext4 changes.

(Sure, there are tons of other changes in between, but they tend to be
ARM changes or drivers that are irrelevant).

It may be that the bad interactive behavior started earlier, but I'd
have expected to notice it - it really is pretty bad, and I've been
doing allmodconfig builds pretty regularly.

(Adding Jan and Fengguang to the cc too, in case it's that
inode_sync_wait() change that *should* be a no-op, but maybe somebody
overlooked something)

NOTE! Latency normally is perfectly fine. Then *occasionally*
something bad happens, and it's horrible for a while. I have no idea
what triggers it. It's a regular "make -j16" of an x86-64 allmodconfig
build, and that should *not* be nearly enough to make this machine
stutter noticeably enough for the mouse pointer to not move smoothly.

I have no real data for this, though - just a "mouse pointer
occasionally freezing up". So I'm looking for a potential "hmm, that
makes me wonder if..." kind of reaction.

                                Linus

Re: GIT PULL] ext4 updates for 3.4

[Linus Torvalds]

On Wed, Mar 28, 2012 at 4:36 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I have no real data for this, though - just a "mouse pointer
> occasionally freezing up". So I'm looking for a potential "hmm, that
> makes me wonder if..." kind of reaction.

Heh. It may have been that my wireless mouse was having some issue
syncing properly with the receiver. I unplugged and replugged the
receiver, and I seem unable to reproduce the problem. So this may have
been just an actual hardware latency issue, rather than the kernel
having bad latency.

But if it made you go "hmm, I should check xyz", please do go ahead
and check it anyway, just in case ;)

                         Linus

Re: GIT PULL] ext4 updates for 3.4

[Fengguang Wu]

On Wed, Mar 28, 2012 at 04:45:55PM -0700, Linus Torvalds wrote:
> On Wed, Mar 28, 2012 at 4:36 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > I have no real data for this, though - just a "mouse pointer
> > occasionally freezing up". So I'm looking for a potential "hmm, that
> > makes me wonder if..." kind of reaction.
> 
> Heh. It may have been that my wireless mouse was having some issue
> syncing properly with the receiver. I unplugged and replugged the
> receiver, and I seem unable to reproduce the problem. So this may have
> been just an actual hardware latency issue, rather than the kernel
> having bad latency.

Yeah my wireless mouse also becomes insensitive nowadays (perhaps
running low on battery power?)

> But if it made you go "hmm, I should check xyz", please do go ahead
> and check it anyway, just in case ;)

I just booted into the new kernel, did some compile testing and it looks good.

Anyway the classical "mouse pointer freezing up" problems are often
caused by waits in direct page reclaim and can be confirmed with large

allocstall      # the number of direct reclaims
nr_vmscan_write # the number of pageout()s

numbers in /proc/vmstat


FYI I'm testing desktop responsiveness remotely (as you know it could
be a painful test). The test scheme is to run

        make -j16  (or any other workloads)

as well as a small script to start some X apps and cycle through the
windows. By checking if there are second(s) long gaps in the below
progress messages, I confirm that the desktop is running smoothly.

 time         window title
  157.68    A swapping-desktop-test
  157.73    A *Unsaved Document 1 - gedit
  157.76    A Restore Session - Iceweasel
  157.80    A Sudoku
  157.83    A LibreOffice 3.4
  157.91    A System Settings
  158.01    A urxvt
  158.01    A xeyes
  158.06    A bay:/home/wfg/swapping-desktop-test - ZSH
  158.08    A Xpdf: /usr/share/doc/shared-mime-info/shared-mime-info-spec.pdf
  158.10    A bay:/home/wfg/swapping-desktop-test - ZSH
  158.17    A OpenOffice.org
  158.20    A OpenOffice.org
  158.21    A OpenOffice.org
  158.22    A OpenOffice.org
  158.24    A OpenOffice.org
  158.25    A Dictionary
  158.27    A bay:/home/wfg/swapping-desktop-test - ZSH
  158.30    A Tetravex
  158.34    A Chess
  158.37    A System Monitor
  158.41    A Mines
  158.50    A Robots
  158.62    A Iagno
  158.68    A Four-in-a-row
  158.75    A Five or More
  158.79    A Klondike
  158.82    A Klotski
  158.90    A Mahjongg - Easy
  158.93    A Tali
  159.00    A Desktop Help
  159.08    A Home

Thanks,
Fengguang

Re: GIT PULL] ext4 updates for 3.4

[Andi Kleen]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Mon, Mar 26, 2012 at 11:07 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>>
>> Ext4 commits for 3.3 merge window; mostly cleanups and bug fixes
>
> Ted, is there any chance that the dirty handling (or something else)
> has some serious latency bug?

There's at least a serious scalability bug. That new extent cache 
statistic count in the super block is showing up badly on IO profiles
on larger systems.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only