Resolved...ish. was: Re: spurious I/O errors from btrfs...at the caching layer?

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

It seems that the rate of spurious I/O errors varies most according to
the vm.vfs_cache_pressure sysctl.  At '10' the I/O errors occur so often
that building a kernel is impossible.  At '100' I can't reproduce even
a single I/O error.

I guess this is own my fault for using non-default sysctl parameters,
although I wouldn't expect any value of this sysctl to cause these
symptoms... :-P


On Sun, Jan 25, 2015 at 11:50:36AM -0500, Zygo Blaxell wrote:
> On Sat, Jan 24, 2015 at 01:06:01PM -0500, Zygo Blaxell wrote:
> > I am seeing a lot of spurious I/O errors that look like they come from
> > the cache-facing side of btrfs.  While running a heavy load with some
> > extent-sharing (e.g. building 20 Linux kernels at once from source trees
> > copied with 'cp -a --reflink=always'), some files will return spurious
> > EIO on read.  It happens often enough to prevent a Linux kernel build
> > about 1/3 of the time.
> [...]
> > Observed from 3.17..3.18.3.  All filesystems affected use skinny-metadata.
> > No filesystems that are not using skinny-metadata seem to have this
> > problem.
> 
> I ran a test overnight using 3.18.3 on a freshly formatted filesystem with
> no skinny-metadata.
> 
> The test consisted of creating reflink copies of a Linux kernel source
> tree and running kernel builds in each copy simultaneously, like this:
> 
> 	# assume you have a ready-to-build kernel tree in 'linux'
> 	for x in $(seq 1 5); do
> 		cp -a --reflink linux linux-$x
> 	done
> 
> 	# build all the kernels at once
> 	for x in $(seq 1 5); do
> 		(cd linux-$x && make -j10 2>&1 | tee make.log) &
> 	done
> 
> 	wait
> 	# then tail all the make.logs and see how many failed due to
> 	# I/O errors
> 
> Spurious I/O errors occured with as few as two concurrent kernel builds.
> 
> The test machine has 16GB of RAM and the filesystem is also 16GB,
> RAID1 on two spinning disks.
> 



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]