spuious I/O errors from btrfs...at the caching layer?

spuious I/O errors from btrfs...at the caching layer?

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

I am seeing a lot of spurious I/O errors that look like they come from
the cache-facing side of btrfs.  While running a heavy load with some
extent-sharing (e.g. building 20 Linux kernels at once from source trees
copied with 'cp -a --reflink=always'), some files will return spurious
EIO on read.  It happens often enough to prevent a Linux kernel build
about 1/3 of the time.

I believe the I/O errors to be spurious because:

	- there is no kernel message of any kind during the event

	- scrub detects 0 errors

	- device stats report 0 errors

	- the drive firmware reports nothing wrong through SMART

	- there seems to be no attempt to read the disk when the error
	is reported

	- "sysctl vm.drop_caches={1,2}" makes the I/O error go away.

Files become unreadable at random, and stay unreadable indefinitely;
however, any time I discover a file that gives EIO on read, I can
poke vm.drop_caches and make the EIO go away.  The file can then be
read normally and has correct contents.  The disk does not seem to be
involved in the I/O error return.

This seems to happen more often when snapshots are being deleted;
however, it occurs on systems with no snapshots as well (though
in these cases the system had snapshots in the past).

When a file returns EIO on read, other snapshots of the same file also
return EIO on read.  I have not been able to test whether this affects
reflink copies (clones) as well.

Observed from 3.17..3.18.3.  All filesystems affected use skinny-metadata.
No filesystems that are not using skinny-metadata seem to have this
problem.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: spuious I/O errors from btrfs...at the caching layer?

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

On Sat, Jan 24, 2015 at 01:06:01PM -0500, Zygo Blaxell wrote:
> I am seeing a lot of spurious I/O errors that look like they come from
> the cache-facing side of btrfs.  While running a heavy load with some
> extent-sharing (e.g. building 20 Linux kernels at once from source trees
> copied with 'cp -a --reflink=always'), some files will return spurious
> EIO on read.  It happens often enough to prevent a Linux kernel build
> about 1/3 of the time.
[...]
> Observed from 3.17..3.18.3.  All filesystems affected use skinny-metadata.
> No filesystems that are not using skinny-metadata seem to have this
> problem.

I ran a test overnight using 3.18.3 on a freshly formatted filesystem with
no skinny-metadata.

The test consisted of creating reflink copies of a Linux kernel source
tree and running kernel builds in each copy simultaneously, like this:

	# assume you have a ready-to-build kernel tree in 'linux'
	for x in $(seq 1 5); do
		cp -a --reflink linux linux-$x
	done

	# build all the kernels at once
	for x in $(seq 1 5); do
		(cd linux-$x && make -j10 2>&1 | tee make.log) &
	done

	wait
	# then tail all the make.logs and see how many failed due to
	# I/O errors

Spurious I/O errors occured with as few as two concurrent kernel builds.

The test machine has 16GB of RAM and the filesystem is also 16GB,
RAID1 on two spinning disks.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Resolved...ish. was: Re: spurious I/O errors from btrfs...at the caching layer?

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

It seems that the rate of spurious I/O errors varies most according to
the vm.vfs_cache_pressure sysctl.  At '10' the I/O errors occur so often
that building a kernel is impossible.  At '100' I can't reproduce even
a single I/O error.

I guess this is own my fault for using non-default sysctl parameters,
although I wouldn't expect any value of this sysctl to cause these
symptoms... :-P


On Sun, Jan 25, 2015 at 11:50:36AM -0500, Zygo Blaxell wrote:
> On Sat, Jan 24, 2015 at 01:06:01PM -0500, Zygo Blaxell wrote:
> > I am seeing a lot of spurious I/O errors that look like they come from
> > the cache-facing side of btrfs.  While running a heavy load with some
> > extent-sharing (e.g. building 20 Linux kernels at once from source trees
> > copied with 'cp -a --reflink=always'), some files will return spurious
> > EIO on read.  It happens often enough to prevent a Linux kernel build
> > about 1/3 of the time.
> [...]
> > Observed from 3.17..3.18.3.  All filesystems affected use skinny-metadata.
> > No filesystems that are not using skinny-metadata seem to have this
> > problem.
> 
> I ran a test overnight using 3.18.3 on a freshly formatted filesystem with
> no skinny-metadata.
> 
> The test consisted of creating reflink copies of a Linux kernel source
> tree and running kernel builds in each copy simultaneously, like this:
> 
> 	# assume you have a ready-to-build kernel tree in 'linux'
> 	for x in $(seq 1 5); do
> 		cp -a --reflink linux linux-$x
> 	done
> 
> 	# build all the kernels at once
> 	for x in $(seq 1 5); do
> 		(cd linux-$x && make -j10 2>&1 | tee make.log) &
> 	done
> 
> 	wait
> 	# then tail all the make.logs and see how many failed due to
> 	# I/O errors
> 
> Spurious I/O errors occured with as few as two concurrent kernel builds.
> 
> The test machine has 16GB of RAM and the filesystem is also 16GB,
> RAID1 on two spinning disks.
> 



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Resolved...ish. was: Re: spurious I/O errors from btrfs...at the caching layer?

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

On 2015-01-25 23:22, Zygo Blaxell wrote:
> It seems that the rate of spurious I/O errors varies most according to
> the vm.vfs_cache_pressure sysctl.  At '10' the I/O errors occur so often
> that building a kernel is impossible.  At '100' I can't reproduce even
> a single I/O error.
>
> I guess this is own my fault for using non-default sysctl parameters,
> although I wouldn't expect any value of this sysctl to cause these
> symptoms... :-P
>
>
Setting that to anything less than about 75 is just asking for trouble. 
  What's I think is happening is that you are eating up memory with the 
vfs cache (because that caches inodes and dentries, which in turn means 
that reflinks don't help decrease it's usage at all) faster than it can 
be reclaimed, and therefore have no space for the file data to be read 
into memory.  This will probably happen with most other filesystems as 
well if you set vfs_cache_pressure really low.

Personally, I wouldn't recommend ever touching that particular sysctl, 
with the exception of increasing it some when you have really fast 
storage (NVMe or UFS-SCSI based SSD's for example) and a relatively 
small set of files that you actually access frequently.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]