Remove the xfs_refcache, it was only needed while we were still building for
2.4 kernels.