[GIT PULL] Btrfs updates

[GIT PULL] Btrfs updates

[Chris Mason]

Hi everyone,

The master branch of the btrfs-unstable tree:

git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git master

Has the first round of btrfs updates for 2.6.35-rc.  I still have some
pending fixes and patches queued up from other people, but this pull
request has the two major features for this release.

First we've got a large ENOSPC improvement from Yan Zheng.  He hit on
all the difficult corners including space balancing, drive management,
fsync logging and many others.  This has been in the works for sometime
and it has held up very well in testing.

The second batch is O_DIRECT work from Josef Bacik.  We've had a few
different iterations on ENOSPC floating on the btrfs list, and this one
takes a much less complex road by hooking into the generic
fs/direct-io.c code.  Josef added a hook for bio submission to
fs/direct-io.c so that we could do proper block number transaction and
properly do multi-device IO.

There are small changes in here to fs/direct-io.c, mm/filemap.c and
include/linux/fs.h.  These have been sent out for review a few times and
commented on.  The filemap.c change allows O_DIRECT reads to fall back
to buffered IO, which we need to support compression and do checksumming
IOs < 1 page.

My commits here are just integrating the two and fixing a few bugs in
the result.

Zheng Yan (13) commits (+4076/-2679):
    Btrfs: Integrate metadata reservation with start_transaction (+678/-528)
    Btrfs: Update metadata reservation for delayed allocation (+232/-415)
    Btrfs: Shrink delay allocated space in a synchronized (+88/-121)
    Btrfs: Introduce contexts for metadata reservation (+853/-385)
    Btrfs: Link block groups of different raid types (+121/-55)
    Btrfs: Metadata ENOSPC handling for balance (+1163/-747)
    Btrfs: Metadata ENOSPC handling for tree log (+156/-128)
    Btrfs: Metadata reservation for orphan inodes (+365/-66)
    Btrfs: Pre-allocate space for data relocation (+92/-45)
    Btrfs: Introduce global metadata reservation (+241/-76)
    Btrfs: Fix block generation verification race (+1/-1)
    Btrfs: Kill allocate_wait in space_info (+58/-76)
    Btrfs: Kill init_btrfs_i() (+28/-36)

Chris Mason (9) commits (+314/-80):
    Btrfs: fix preallocation and nodatacow checks in O_DIRECT (+140/-16)
    Btrfs: move O_DIRECT space reservation to btrfs_direct_IO (+8/-6)
    Btrfs: don't walk around with task->state != TASK_RUNNING (+4/-3)
    Btrfs: use async helpers for DIO write checksumming (+52/-17)
    Btrfs: add more error checking to btrfs_dirty_inode (+13/-2)
    Btrfs: avoid ENOSPC errors in btrfs_dirty_inode (+11/-3)
    Btrfs: rework O_DIRECT enospc handling (+49/-30)
    Btrfs: drop verbose enospc printk (+2/-0)
    Btrfs: allow unaligned DIO (+35/-3)

Josef Bacik (5) commits (+829/-134):
    direct-io: add a hook for the fs to provide its own submit_bio function (+45/-8)
    fs: allow short direct-io reads to be completed via buffered IO (+31/-5)
    direct-io: do not merge logically non-contiguous requests (+18/-2)
    Btrfs: add basic DIO read/write support (+631/-36)
    Btrfs: do aio_write instead of write (+104/-83)

Total: (27) commits

 fs/btrfs/extent-tree.c  | 2317 ++++++++++++++++++++++++++++--------------------
 fs/btrfs/relocation.c   | 1991 +++++++++++++++++++++++++----------------
 fs/btrfs/inode.c        | 1797 +++++++++++++++++++++++++++++--------
 fs/btrfs/file.c         |  304 +++---
 fs/btrfs/tree-log.c     |  241 +++-
 fs/btrfs/transaction.c  |  232 +++-
 fs/btrfs/ioctl.c        |  206 ++--
 fs/btrfs/disk-io.c      |  171 +--
 fs/btrfs/ctree.h        |  163 ++-
 fs/btrfs/ctree.c        |  109 +-
 fs/btrfs/delayed-ref.c  |  101 --
 fs/btrfs/extent_io.c    |   85 -
 fs/btrfs/ordered-data.c |   82 +
 fs/direct-io.c          |   62 +
 mm/filemap.c            |   36 
 fs/btrfs/super.c        |   30 
 fs/btrfs/file-item.c    |   28 
 fs/btrfs/inode-item.c   |   27 
 fs/btrfs/transaction.h  |   24 
 fs/btrfs/root-tree.c    |   23 
 fs/btrfs/volumes.c      |   17 
 fs/btrfs/extent_io.h    |   14 
 fs/btrfs/xattr.c        |   12 
 include/linux/fs.h      |   11 
 fs/btrfs/ordered-data.h |    9 
 fs/btrfs/tree-defrag.c  |    7 
 fs/btrfs/disk-io.h      |    4 
 fs/btrfs/delayed-ref.h  |    3 
 fs/btrfs/btrfs_inode.h  |    3 
 fs/btrfs/tree-log.h     |    2 
 fs/btrfs/async-thread.c |    1 
 31 files changed, 5219 insertions(+), 2893 deletions(-)

Re: [GIT PULL] Btrfs updates

[Linus Torvalds]

On Thu, 27 May 2010, Chris Mason wrote:
> 
>  fs/btrfs/extent-tree.c  | 2317 ++++++++++++++++++++++++++++--------------------
>  fs/btrfs/relocation.c   | 1991 +++++++++++++++++++++++++----------------
>  fs/btrfs/inode.c        | 1797 +++++++++++++++++++++++++++++--------
>  fs/btrfs/file.c         |  304 +++---
>  fs/btrfs/tree-log.c     |  241 +++-
> ...

What kind of bogus diffstat is this? 

Don't do that. I cannot compare your bogus diffstat with what I get, 
because it's just random noise. You've apparently sorted it by size of the 
damage, but the numbers are total crap too. That is _not_ the actual size 
of the changes at all.

I suspect you have some script that adds up all the patches, but that's 
wrong. If a subsequent patch changes the things that an earlier patch has 
done, then the numbers don't just add up.

And if the diffstat doesn't match what I get when I pull, then my reaction 
inevitably is "ok, that's not what they asked me to pull, so I'll just 
reject it out-of-hand". So don't play games with diffstats - that just 
means that things won't get pulled.

		Linus

Re: [GIT PULL] Btrfs updates

[Chris Mason]

On Thu, May 27, 2010 at 10:18:04AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 27 May 2010, Chris Mason wrote:
> > 
> >  fs/btrfs/extent-tree.c  | 2317 ++++++++++++++++++++++++++++--------------------
> >  fs/btrfs/relocation.c   | 1991 +++++++++++++++++++++++++----------------
> >  fs/btrfs/inode.c        | 1797 +++++++++++++++++++++++++++++--------
> >  fs/btrfs/file.c         |  304 +++---
> >  fs/btrfs/tree-log.c     |  241 +++-
> > ...
> 
> What kind of bogus diffstat is this? 
> 
> Don't do that. I cannot compare your bogus diffstat with what I get, 
> because it's just random noise. You've apparently sorted it by size of the 
> damage, but the numbers are total crap too. That is _not_ the actual size 
> of the changes at all.

My the btrfs-unstable repo is against v2.6.34, here is a 'pure'
diffstat.  My script ends up piping the git log -p output through
diffstat, which does give different numbers.  I'll use the git diff
variant from now on.

# git diff v2.6.34 HEAD | diffstat
 fs/btrfs/async-thread.c |    1 
 fs/btrfs/btrfs_inode.h  |    3 
 fs/btrfs/ctree.c        |  109 +-
 fs/btrfs/ctree.h        |  163 ++-
 fs/btrfs/delayed-ref.c  |  101 --
 fs/btrfs/delayed-ref.h  |    3 
 fs/btrfs/disk-io.c      |  169 +--
 fs/btrfs/disk-io.h      |    4 
 fs/btrfs/extent-tree.c  | 2255 ++++++++++++++++++++++++++++--------------------
 fs/btrfs/extent_io.c    |   85 -
 fs/btrfs/extent_io.h    |   14 
 fs/btrfs/file-item.c    |   28 
 fs/btrfs/file.c         |  166 ++-
 fs/btrfs/inode-item.c   |   27 
 fs/btrfs/inode.c        | 1713 ++++++++++++++++++++++++++++--------
 fs/btrfs/ioctl.c        |  206 ++--
 fs/btrfs/ordered-data.c |   82 +
 fs/btrfs/ordered-data.h |    9 
 fs/btrfs/relocation.c   | 1971 +++++++++++++++++++++++++----------------
 fs/btrfs/root-tree.c    |   23 
 fs/btrfs/super.c        |   30 
 fs/btrfs/transaction.c  |  232 +++-
 fs/btrfs/transaction.h  |   24 
 fs/btrfs/tree-defrag.c  |    7 
 fs/btrfs/tree-log.c     |  241 +++--
 fs/btrfs/tree-log.h     |    2 
 fs/btrfs/volumes.c      |   17 
 fs/btrfs/xattr.c        |   12 
 fs/direct-io.c          |   62 +
 include/linux/fs.h      |   11 
 mm/filemap.c            |   36 
 31 files changed, 5066 insertions(+), 2740 deletions(-)

-chris

Re: [GIT PULL] Btrfs updates

[Linus Torvalds]

On Thu, 27 May 2010, Chris Mason wrote:
> 
> # git diff v2.6.34 HEAD | diffstat

That still has the potential to be wrong (but got the numbers I expected 
this time). It will be wrong in several cases:

 - "diffstat" has some random common prefix removal logic that I've never 
   figured out the exact rules for.

   You don't happen to see it, because you actually had changes outside of 
   fs/btrfs (so there was no common prefix to worry about), but if 
   everything had been inside fs/btrfs, then at least some versions of 
   diffstat will just remove that whole thing as common, and talk about 
   changes to 'ioctl.c' rather than 'fs/btrfs/ioctl.c'

   (And no, I'm not entirely sure what triggers it. It might only happen 
   for certain patterns and/or certain versions of diffstat, but it's a 
   reason to avoid diffstat in general, or at least use "-p1" to make it 
   reliable)

 - it will do the wrong thing for renames and copies.

 - it doesn't give the summary that pull-request does, which talks about 
   new, deleted and file-mode-changed files.

So just do

	git diff --stat --summary -M v2.6.34..HEAD

instead, which gets all the above cases right. Also, you don't even need 
to remember where you started - you might as well use git to do that too, 
and write it as (assuming you have an 'origin' branch that points to the 
upstream tree):

	git diff --stat --summary -M origin...HEAD

(note the *three* dots: that says that you should diff against the common 
ancestor, so git will just figure out where you started on its own).

		Linus

Re: [GIT PULL] Btrfs updates

[Miao Xie]

Hi, Chris

on 2010-5-27 23:15, Chris Mason wrote: 
> My commits here are just integrating the two and fixing a few bugs in
> the result.
> 
> Zheng Yan (13) commits (+4076/-2679):
>     Btrfs: Integrate metadata reservation with start_transaction (+678/-528)
>     Btrfs: Update metadata reservation for delayed allocation (+232/-415)
>     Btrfs: Shrink delay allocated space in a synchronized (+88/-121)
>     Btrfs: Introduce contexts for metadata reservation (+853/-385)
>     Btrfs: Link block groups of different raid types (+121/-55)
>     Btrfs: Metadata ENOSPC handling for balance (+1163/-747)
>     Btrfs: Metadata ENOSPC handling for tree log (+156/-128)
>     Btrfs: Metadata reservation for orphan inodes (+365/-66)
>     Btrfs: Pre-allocate space for data relocation (+92/-45)
>     Btrfs: Introduce global metadata reservation (+241/-76)
>     Btrfs: Fix block generation verification race (+1/-1)
>     Btrfs: Kill allocate_wait in space_info (+58/-76)
>     Btrfs: Kill init_btrfs_i() (+28/-36)
> 
> Chris Mason (9) commits (+314/-80):
>     Btrfs: fix preallocation and nodatacow checks in O_DIRECT (+140/-16)
>     Btrfs: move O_DIRECT space reservation to btrfs_direct_IO (+8/-6)
>     Btrfs: don't walk around with task->state != TASK_RUNNING (+4/-3)
>     Btrfs: use async helpers for DIO write checksumming (+52/-17)
>     Btrfs: add more error checking to btrfs_dirty_inode (+13/-2)
>     Btrfs: avoid ENOSPC errors in btrfs_dirty_inode (+11/-3)
>     Btrfs: rework O_DIRECT enospc handling (+49/-30)
>     Btrfs: drop verbose enospc printk (+2/-0)
>     Btrfs: allow unaligned DIO (+35/-3)
> 
> Josef Bacik (5) commits (+829/-134):
>     direct-io: add a hook for the fs to provide its own submit_bio function (+45/-8)
>     fs: allow short direct-io reads to be completed via buffered IO (+31/-5)
>     direct-io: do not merge logically non-contiguous requests (+18/-2)
>     Btrfs: add basic DIO read/write support (+631/-36)
>     Btrfs: do aio_write instead of write (+104/-83)
> 
> Total: (27) commits
> 

In the following mail, You said that you had put the cleanups which Zhao Lei and I contributed
into a .35 queue, but I cannot find those patches have been merged into the .35 kernel...
  http://marc.info/?l=linux-btrfs&m=127049622906168&w=2

Is there anything wrong with those patches?

Regards
Miao

Re: [GIT PULL] Btrfs updates

[Felipe Contreras]

On Thu, May 27, 2010 at 6:15 PM, Chris Mason <chris.mason@oracle.com> wrote:
>    Btrfs: Fix block generation verification race (+1/-1)

This seems to cause a hang when there are "parent transid verify failed" errors.

Somebody on #btrfs told me to revert it, and at least the system
doesn't hang any more.

-- 
Felipe Contreras
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html