[ANNOUNCE] xfsprogs: v3.2.0 released!

[ANNOUNCE] xfsprogs: v3.2.0 released!

[Dave Chinner]

[-- Attachment #1.1: Type: text/plain, Size: 6221 bytes --]

Hello!

It is my pleasure to announce the release of v3.2.0 of the xfsprogs
package. This release has been a long time in the making with a lot
of effort from many people:

	- just over a year since v3.1.11 was released
	- 323 commits
	- 17 contributors
	- 10 code reviewers
	- 232 files changed
	- +29549/-24432 lines of code

The major feature that this release brings is support for the
version 5 on-disk format. This new format provides significant
reliability enhancements such as metadata CRCs, object back owner
references and improved crash recovery reliability.

The next kernel to be released (v3.15) will also be the first kernel
to support this new on-disk format for production workloads: the
kernel no longer considers this new format experimental as of
v3.15-rc5.

There are many other enhancements:

	- xfs_repair has significantly improved multithreading
	  scalability
	- xfs_db now uses the common libxfs IO engine
	- xfs_db validates the metadata it reads from disk
	- xfs_io has support for a wide range of new kernel features
	- xfs_metadump has much improved obfuscation support
	- libxfs has significanty fewer differences to the kernel
	  code it is derived from

The source code can be accessed via git using this URL:

	git://oss.sgi.com/xfs/cmds/xfsprogs.git

A signed gzipped-tar archive of the source code is available here:

	ftp://oss.sgi.com/projects/xfs/cmd_tars/xfsprogs-3.2.0.tar.gz
	ftp://oss.sgi.com/projects/xfs/cmd_tars/xfsprogs-3.2.0.tar.gz.sign

Problems, issues, questions and general discussion about the release
should be directed to the XFS mailing list (xfs@oss.sgi.com).

A summary of the changes during development of this release (taken
from doc/CHANGES) is as follows:

xfsprogs-3.2.0 (16 May 2014)
        - First release with full support of CRC enabled filesystems
        - No code changes from 3.2.0-rc3

xfsprogs-3.2.0-rc3 (9 May 2014)
        - Third release candidate for full support of CRC enabled filesystems
        - Updated Debian change logs in preparation for release (Nathan Scott)
        - Build warning fixes (Nathan Scott)
        - xfs_repair prefetch fix (Eric Sandeen)
        - xfs_repair block tracking scalability fix

xfsprogs-3.2.0-rc2 (2 May 2014)
        - Second release candidate for full support of CRC enabled filesystems
        - xfs_repair has full CRC validation and repair
        - Coverity related cleanups and fixes

xfsprogs-3.2.0-rc1 (14 April 2014)
        - First release candidate for full support of CRC enabled filesystems
        - Large number of Coverity related fixes and cleanups
        - disambiguous of CRC validation errors from IO errors.
        - Improved dangerous mode handling in repair
        - repair handles garbage in zeroed areas of superblocks better
        - repair validates dirent ftype field fully
        - metadump fully supports discontiguous directory blocks
        - metadump only recalculates CRCs on metadata it obfuscates so as to
          preserve errors in the metadata where possible.
        - default log size that mkfs creates is now reverted to the same size
          as 3.1.x releases create.
        - mkfs sets the ftype on directory entries correctly during protofile
          population
        - xfs_io support O_TMPFILE, flink, FALLOC_FL_ZERO_RANGE and
          FALLOC_FL_COLLAPSE_RANGE,
        - logprint handles split entries better

xfsprogs-3.2.0-alpha2 (25 November 2013)
        - Alpha release for the purpose of testing the CRC feature in
          kernels 3.10 and newer.
        - Enable xfs_db write support and xfs_metadump support for CRC
          enabled filesystems.
        - Add directory entry filetype support for non-CRC filesystems.
        - Remove experimental warnings for CRC filesystems.
        - Ensure all inodes created by xfs_repair have a proper d_type set.
        - Fix build on big endian machines.
        - Properly handle symlinks to devices on various tool commandlines.
        - Fix xfs_repair's dirty log detection for 4k sector logs, broken
          in Alpha1.
        - Fix a potential segfault in xfs_repair when issuing progress
          reports.
        - Fix potential xfs_fsr failures when running w/ selinux.
        - Update config.guess/config.sub for arm64, thanks to Colin Watson.
        - Stop wasting memory by caching inode structures in xfs_repair -
          they are never re-used.  Thanks to Christoph Hellwig.
        - Fix several Coverity-found defects, thanks to Li Zhong.
        - Fix platform_test_xfs_fd to return false on special files which
          cannot take an xfs ioctl.
        - Sync up libxfs with kernel code.
        - Improved xfs_repair performance on large filesystems
          (always use prefetch and strided AG scanning functionality)


xfsprogs-3.2.0-alpha1 (26 September 2013)
        - Alpha release for the purpose of testing the CRC feature in
          kernels 3.10 and newer.
        - Remove all vestiges of old, unsupported version 1 directory code.
        - Add a "readdir" command to xfs_io, thanks to Brian Foster.
        - Fix potential segfault in xfs_repair when creating lost+found.
        - Zero out unused parts of on-disk superblocks during repair, to
          avoid metadata verifier failures at runtime.
        - Add directory entry type support to mkfs.xfs and xfs_db.
        - Add the icreate transaction to xfs_logprint, and fix continuation
          transactions.
        - Add the lseek SEEK_DATA/SEEK_HOLE support into xfs_io.
        - Print all AGI unlinked buckets in xfs_logprint.
        - Fix mkfs.xfs ENOSPC with protofile which creates a very large
          directory.
        - Fix several Coverity-found defects, thanks to Li Zhong.
        - Do all file reads in xfs_fsr using O_DIRECT.
        - Sync up libxfs with kernel code.
        - Add support for concurrent group and project quota usage on CRC
          enabled filesystems.
        - Ensure mkfs creates log sizes that are always large enough for
          the configured fileystem geometry.

-- 
Dave Chinner
david@fromorbit.com

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ANNOUNCE] xfsprogs: v3.2.0 released!

[Adam Sampson]

Dave Chinner <david@fromorbit.com> writes:

> It is my pleasure to announce the release of v3.2.0 of the xfsprogs
> package.

If this is built with DEBUG= (i.e. not defaulting to DEBUG=-NDEBUG),
several source files fail to compile -- it looks like there are a number
of assertions that haven't been updated for changes in the code:

logitem.c:53:24: error: 'struct xfs_buf' has no member named 'b_map_count'
    ASSERT(blip->bli_buf->b_map_count == nmaps);
                        ^
trans.c:52:12: error: 'struct xfs_log_item' has no member named 'li_ailp'
  ASSERT(lip->li_ailp == tp->t_mountp->m_ail);
            ^
trans.c:52:37: error: 'xfs_mount_t' has no member named 'm_ail'
  ASSERT(lip->li_ailp == tp->t_mountp->m_ail);
                                     ^
util.c:253:40: error: 'mp' undeclared (first use in this function)
   ASSERT(uuid_equal(&ip->i_d.di_uuid, &mp->m_sb.sb_uuid));
                                        ^
util.c:401:33: error: 'xfs_ifork_t' has no member named 'if_ext_max'
   ip->i_d.di_nextents > ip->i_df.if_ext_max);
                                 ^
xfs_btree.c:545:2: warning: implicit declaration of function 'xfs_buf_geterror' [-Wimplicit-function-declaration]
  ASSERT(!xfs_buf_geterror(bp));
  ^
xfs_btree.c:2052:21: error: 'const struct xfs_btree_ops' has no member named 'keys_inorder'
   ASSERT(cur->bc_ops->keys_inorder(cur,
                     ^
xfs_btree.c:2064:21: error: 'const struct xfs_btree_ops' has no member named 'recs_inorder'
   ASSERT(cur->bc_ops->recs_inorder(cur,
                     ^
xfs_btree.c:2240:21: error: 'const struct xfs_btree_ops' has no member named 'keys_inorder'
   ASSERT(cur->bc_ops->keys_inorder(cur, rkp,
                     ^
xfs_btree.c:2259:21: error: 'const struct xfs_btree_ops' has no member named 'recs_inorder'
   ASSERT(cur->bc_ops->recs_inorder(cur, rrp,
                     ^
xfs_dir2_node.c:1053:36: error: 'oldstale' undeclared (first use in this function)
  ASSERT(hdr1.stale + hdr2.stale == oldstale);
                                    ^
xfs_inode_fork.c:728:2: warning: implicit declaration of function 'xfs_isilocked' [-Wimplicit-function-declaration]
  ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
  ^
xfs_inode_fork.c:728:42: error: 'XFS_ILOCK_SHARED' undeclared (first use in this function)
  ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
                                          ^

(Errors from current Git, built with GCC 4.9. You also get a good crop
of warnings if you build it with clang 3.4.1's scan-build, which'd be
worth checking out in case there's anything serious there.)

> A signed gzipped-tar archive of the source code is available here:

It's signed with a different key from the previous release -- it'd be
useful to mention the key ID in the announcement so that it can be
verified.

Cheers,

-- 
Adam Sampson <ats@offog.org>                         <http://offog.org/>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ANNOUNCE] xfsprogs: v3.2.0 released!

[Dave Chinner]

On Sat, May 17, 2014 at 12:02:15AM +0100, Adam Sampson wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > It is my pleasure to announce the release of v3.2.0 of the xfsprogs
> > package.
> 
> If this is built with DEBUG= (i.e. not defaulting to DEBUG=-NDEBUG),
> several source files fail to compile -- it looks like there are a number
> of assertions that haven't been updated for changes in the code:

Can't say I've ever built xfsprogs with "DEBUG=". I'm not sure
there's really any benefit in doing so - it's preferable to have
things like xfs_repair abort when it comes across an inconsistency
it can't handle than to continue blindly along and making a bigger
mess of the filesystem it's supposed to be fixing...

Anyway, we'll look to fix it for 3.2.1.

> (Errors from current Git, built with GCC 4.9. You also get a good crop
> of warnings if you build it with clang 3.4.1's scan-build, which'd be
> worth checking out in case there's anything serious there.)

ISTR that was done recently by Eric, and I've run clang recently,
too.

> > A signed gzipped-tar archive of the source code is available here:
> 
> It's signed with a different key from the previous release -- it'd be
> useful to mention the key ID in the announcement so that it can be
> verified.

Should be obvious, yes? Especially with the release announcement
being signed, too. As it is, release tarballs have always been signed
with the current maintainer's key, so when the maintainer changes so
does the signing key....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ANNOUNCE] xfsprogs: v3.2.0 released!

[Eric Sandeen]

On 5/16/14, 7:23 PM, Dave Chinner wrote:
> On Sat, May 17, 2014 at 12:02:15AM +0100, Adam Sampson wrote:
>> Dave Chinner <david@fromorbit.com> writes:
>>
>>> It is my pleasure to announce the release of v3.2.0 of the xfsprogs
>>> package.
>>
>> If this is built with DEBUG= (i.e. not defaulting to DEBUG=-NDEBUG),
>> several source files fail to compile -- it looks like there are a number
>> of assertions that haven't been updated for changes in the code:
> 
> Can't say I've ever built xfsprogs with "DEBUG=". I'm not sure
> there's really any benefit in doing so - it's preferable to have
> things like xfs_repair abort when it comes across an inconsistency
> it can't handle than to continue blindly along and making a bigger
> mess of the filesystem it's supposed to be fixing...
> 
> Anyway, we'll look to fix it for 3.2.1.

-NDEBUG is only default for libxfs/ AFAIK:

# grep -B1 NDEBUG libxfs/Makefile 
# don't try linking xfs_repair with a debug libxfs.
DEBUG = -DNDEBUG

For everything else, default should be -DDEBUG:

configure:    DEBUG=${DEBUG:-'-DDEBUG'}		    debug_build="$DEBUG"

so default is -DDEBUG for everything except libxfs, with -NDEBUG;
I'm not sure what '' does - break, apparently.  Seems like maybe just
a bit of a makefile mess and maybe old crufty code.
Patches accepted and all that.  :)

But maybe it's time to remove it as an option, and just DTRT everywhere.

>> (Errors from current Git, built with GCC 4.9. You also get a good crop
>> of warnings if you build it with clang 3.4.1's scan-build, which'd be
>> worth checking out in case there's anything serious there.)
> 
> ISTR that was done recently by Eric, and I've run clang recently,
> too.

Some, yeah.  I don't claim to have gotten it all cleaned up yet,
though.  We also keep track of what Coverity finds, chipping away
at a backlog of potential defects...

Thanks for the report, though, there's always more to do...

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ANNOUNCE] xfsprogs: v3.2.0 released!

[Adam Sampson]

Dave Chinner <david@fromorbit.com> writes:

> [...] it's preferable to have things like xfs_repair abort when it
> comes across an inconsistency it can't handle than to continue blindly
> along and making a bigger mess of the filesystem it's supposed to be
> fixing...

Yes -- that's why I was building with DEBUG= on previous releases
(i.e. I want assertions enabled). doc/INSTALL says that DEBUG=-DNDEBUG
disables assertions, so packagers are quite likely to have DEBUG= in
their build process.

> Anyway, we'll look to fix it for 3.2.1.

Cheers. :)

-- 
Adam Sampson <ats@offog.org>                         <http://offog.org/>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ANNOUNCE] xfsprogs: v3.2.0 released!

[Dave Chinner]

On Sat, May 17, 2014 at 11:53:08AM +0100, Adam Sampson wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > [...] it's preferable to have things like xfs_repair abort when it
> > comes across an inconsistency it can't handle than to continue blindly
> > along and making a bigger mess of the filesystem it's supposed to be
> > fixing...
> 
> Yes -- that's why I was building with DEBUG= on previous releases
> (i.e. I want assertions enabled). doc/INSTALL says that DEBUG=-DNDEBUG
> disables assertions, so packagers are quite likely to have DEBUG= in
> their build process.

Hmmm - so, not being an everyday userspace programmer, it didn't
even occur to me that "-DNDEBUG" actually changes libc header
behaviour, not anything to do with the XFS code.

$ man assert
.....
BUGS
       assert()  is  implemented as a macro; if the expression
       tested has side-effects, program behavior will be different
       depending on whether NDEBUG is defined.  This may create
       Heisenbugs which go away when debugging is turned on.

Yup, it's oh so obvious now that "NDEBUG" is something owned by
system library code, not the xfsprogs package...

<sigh>

> > Anyway, we'll look to fix it for 3.2.1.

Or maybe not. The intent of always turning off the asserts is that
code like xfs_repair shouldn't assert fail when stuff it detected as
out of bounds in a library function. IOWs, you're quite likely to
unintentionally break repair by removing the NDEBUG define to
re-instate the library asserts...

The control of the assert statements in the xfs_repair code itself
is handled by the -DDEBUG macro, which is configurable. i.e. you can
chose whether or not to have asserts in the repair code itself for
it to fail when an inconsistency it can't handle is detected, but
repair defines "inconsistent and cannot continue" very differently
to the libxfs library code....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ANNOUNCE] xfsprogs: v3.2.0 released!

[Dave Chinner]

On Mon, May 19, 2014 at 12:55:53PM +1000, Dave Chinner wrote:
> On Sat, May 17, 2014 at 11:53:08AM +0100, Adam Sampson wrote:
> > Dave Chinner <david@fromorbit.com> writes:
> > 
> > > [...] it's preferable to have things like xfs_repair abort when it
> > > comes across an inconsistency it can't handle than to continue blindly
> > > along and making a bigger mess of the filesystem it's supposed to be
> > > fixing...
> > 
> > Yes -- that's why I was building with DEBUG= on previous releases
> > (i.e. I want assertions enabled). doc/INSTALL says that DEBUG=-DNDEBUG
> > disables assertions, so packagers are quite likely to have DEBUG= in
> > their build process.
> 
> Hmmm - so, not being an everyday userspace programmer, it didn't
> even occur to me that "-DNDEBUG" actually changes libc header
> behaviour, not anything to do with the XFS code.
> 
> $ man assert
> .....
> BUGS
>        assert()  is  implemented as a macro; if the expression
>        tested has side-effects, program behavior will be different
>        depending on whether NDEBUG is defined.  This may create
>        Heisenbugs which go away when debugging is turned on.
> 
> Yup, it's oh so obvious now that "NDEBUG" is something owned by
> system library code, not the xfsprogs package...
> 
> <sigh>
> 
> > > Anyway, we'll look to fix it for 3.2.1.
> 
> Or maybe not. The intent of always turning off the asserts is that
> code like xfs_repair shouldn't assert fail when stuff it detected as
> out of bounds in a library function. IOWs, you're quite likely to
> unintentionally break repair by removing the NDEBUG define to
> re-instate the library asserts...

And it's not straight forward, either, because some of the ASSERT()s
have code in them that is only present when the libxfs code is built
with -DDEBUG, so compiling in the asserts without enabling -DDEBUG is
not going to work. And, well, compiling with -DDEBUG itself deosn't
work nor with -DNDEBUG -DDEBUG because -DDEBUG enables code that has
unhandled kernelisms in it.

So, right now, don't remove -NDEBUG from the libxfs/Makefile. It's
there for a good reason right now. i.e. removing it requires some
non-trivial work, and even if we do make it compile the resultant
code will probably not work (i.e. assert fail unnecessarily) given
the requirements that userspace has for tolerance of inconsistent
on-disk states.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs