BTRFS: error (device md1) in btrfs_commit_transaction:2371: errno=-5 IO failure

BTRFS: error (device md1) in btrfs_commit_transaction:2371: errno=-5 IO failure

[Marian Popeanga]

Hi All, 

I am getting this error on a nas server using 4 HDD, recently after i've had disks in a raid 5 array expanded in size and the file system expanded as well.
What to point out, that during the sync process when i replaced one of the disk, there was other disk which complained that there is 1 byte error (s.m.a.r.t reported it)on it which can not relocated or read.
Thats 1 byte error looks to have been there before, but it never caused a problem, like that i am seeing now. 

That disk itself never failed during the sync and replacement of all the other disks. It just had these IO error probably related to that 1 byte error, which cause the nas to actually flag me about this problem first time.

After  all the 4 disks were upgraded, i did a btrfs check and there wasn't any error reported. Now, this error shows up and makes the file system read only.  I've even tried a btrfs --repair but this error is still there.

I am wondering if this can be fixed without having to rebuild all this file system and what can be the steps? 

Is there a way to know on which access this error happens ? Been trying to read all th files on the file system and they read fine, but this error still showed up.

[29495.068397] BTRFS error (device md1): bdev /dev/md1 errs: wr 11, rd 5, flush 0, corrupt 0, gen 0
[29495.077392] BTRFS error (device md1): bdev /dev/md1 errs: wr 12, rd 5, flush 0, corrupt 0, gen 0
[29495.086360] BTRFS error (device md1): bdev /dev/md1 errs: wr 13, rd 5, flush 0, corrupt 0, gen 0
[29495.096714] BTRFS: error (device md1) in btrfs_commit_transaction:2371: errno=-5 IO failure (Error while writing out transaction)
[29495.108649] BTRFS info (device md1): forced readonly
[29495.113762] BTRFS warning (device md1): Skipping commit of aborted transaction.
[29495.121259] BTRFS: error (device md1) in cleanup_transaction:1975: errno=-5 IO failure

The kernel version is :  5.13.x x86_64 GNU/Linux

thanks alot