metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Stefan Priebe - Profihost AG]

Hello,

under very high load i'm getting the following error on a bcache device
using XFS:
XFS (bcache1): metadata I/O error: block 0x575b6d70
("xfs_trans_read_buf_map") error 4 numblks 16
XFS (bcache1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -4.
XFS (bcache1): xfs_do_force_shutdown(0x8) called from line 3519 of file
fs/xfs/xfs_inode.c. Return address = 0xffffffff993441ff
XFS (bcache1): Corruption of in-memory data detected. Shutting down
filesystem
XFS (bcache1): Please umount the filesystem and rectify the problem(s)
XFS (bcache1): xfs_log_force: error -5 returned.
XFS (bcache1): xfs_log_force: error -5 returned.
XFS (bcache1): xfs_log_force: error -5 returned.

The used kernel is 4.4.79 based while using an opensuses patchset. Is
this something which is a known problem or something suse has broken?

Thanks!

Greets,
Stefan

Re: metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1462 bytes --]

Le Tue, 1 Aug 2017 10:12:57 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

> Hello,
> 
> under very high load i'm getting the following error on a bcache
> device using XFS:
> XFS (bcache1): metadata I/O error: block 0x575b6d70
> ("xfs_trans_read_buf_map") error 4 numblks 16
> XFS (bcache1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -4.
> XFS (bcache1): xfs_do_force_shutdown(0x8) called from line 3519 of
> file fs/xfs/xfs_inode.c. Return address = 0xffffffff993441ff
> XFS (bcache1): Corruption of in-memory data detected. Shutting down
> filesystem
> XFS (bcache1): Please umount the filesystem and rectify the problem(s)
> XFS (bcache1): xfs_log_force: error -5 returned.
> XFS (bcache1): xfs_log_force: error -5 returned.
> XFS (bcache1): xfs_log_force: error -5 returned.
> 
> The used kernel is 4.4.79 based while using an opensuses patchset. Is
> this something which is a known problem or something suse has broken?

Do you know if the patchset includes Eric Wheeler's stability patches
for bcache?

Also please post the output from dmesg with the complete trace.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Stefan Priebe - Profihost AG]

Hello Emmanuel,

yes there are bcache patches already applied:
0002-bcache-bch_writeback_thread-is-not-freezable.patch
0003-bcache-bch_allocator_thread-is-not-freezable.patch
0001-bcache-partition-support-add-16-minors-per-bcacheN-d.patch
0044-bcache-Remove-deprecated-create_workqueue.patch
0045-bcache-Remove-redundant-parameter-for-cache_alloc.patch
0046-bcache-update-document-info.patch
0047-bcache-Remove-redundant-block_size-assignment.patch
0048-bcache-register_bcache-call-blkdev_put-when-cache_al.patch
0049-bcache-pr_err-more-meaningful-error-message-when-nr_.patch
0071-bcache-switch-to-using-blk_queue_write_cache.patch
0092-bcache-use-op_is_write-instead-of-checking-for-REQ_W.patch
0110-bcache-use-bio-op-accessors.patch
0253-bcache-use-op_is_sync-to-check-for-synchronous-reque.patch
0344-bcache-debug-avoid-accessing-.bi_io_vec-directly.patch
0002-bcache-fix-calling-ida_simple_remove-with-incorrect-minor.patch
0001-bcache-force-trigger-gc.patch
0001-bcache-only-recovery-I-O-error-for-writethrough-mode.patch

There is no trace involved as there is no kernel bug triggered. That's
the whole output.

Greets,
Stefan
Am 01.08.2017 um 15:35 schrieb Emmanuel Florac:
> Le Tue, 1 Aug 2017 10:12:57 +0200
> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:
> 
>> Hello,
>>
>> under very high load i'm getting the following error on a bcache
>> device using XFS:
>> XFS (bcache1): metadata I/O error: block 0x575b6d70
>> ("xfs_trans_read_buf_map") error 4 numblks 16
>> XFS (bcache1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -4.
>> XFS (bcache1): xfs_do_force_shutdown(0x8) called from line 3519 of
>> file fs/xfs/xfs_inode.c. Return address = 0xffffffff993441ff
>> XFS (bcache1): Corruption of in-memory data detected. Shutting down
>> filesystem
>> XFS (bcache1): Please umount the filesystem and rectify the problem(s)
>> XFS (bcache1): xfs_log_force: error -5 returned.
>> XFS (bcache1): xfs_log_force: error -5 returned.
>> XFS (bcache1): xfs_log_force: error -5 returned.
>>
>> The used kernel is 4.4.79 based while using an opensuses patchset. Is
>> this something which is a known problem or something suse has broken?
> 
> Do you know if the patchset includes Eric Wheeler's stability patches
> for bcache?
> 
> Also please post the output from dmesg with the complete trace.
> 

Re: metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]

Le Tue, 1 Aug 2017 15:50:20 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

> Hello Emmanuel,
> 
> yes there are bcache patches already applied:
> 0002-bcache-bch_writeback_thread-is-not-freezable.patch
> 0003-bcache-bch_allocator_thread-is-not-freezable.patch
> 0001-bcache-partition-support-add-16-minors-per-bcacheN-d.patch
> 0044-bcache-Remove-deprecated-create_workqueue.patch
> 0045-bcache-Remove-redundant-parameter-for-cache_alloc.patch
> 0046-bcache-update-document-info.patch
> 0047-bcache-Remove-redundant-block_size-assignment.patch
> 0048-bcache-register_bcache-call-blkdev_put-when-cache_al.patch
> 0049-bcache-pr_err-more-meaningful-error-message-when-nr_.patch
> 0071-bcache-switch-to-using-blk_queue_write_cache.patch
> 0092-bcache-use-op_is_write-instead-of-checking-for-REQ_W.patch
> 0110-bcache-use-bio-op-accessors.patch
> 0253-bcache-use-op_is_sync-to-check-for-synchronous-reque.patch
> 0344-bcache-debug-avoid-accessing-.bi_io_vec-directly.patch
> 0002-bcache-fix-calling-ida_simple_remove-with-incorrect-minor.patch
> 0001-bcache-force-trigger-gc.patch
> 0001-bcache-only-recovery-I-O-error-for-writethrough-mode.patch

Good.
 
> There is no trace involved as there is no kernel bug triggered. That's
> the whole output.

Too bad :/ What are the underlying devices? No error at all at the
kernel level? No IO error? Is bcache configured write-back or
write-through?

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Dave Chinner]

On Tue, Aug 01, 2017 at 10:12:57AM +0200, Stefan Priebe - Profihost AG wrote:
> Hello,
> 
> under very high load i'm getting the following error on a bcache device
> using XFS:
> XFS (bcache1): metadata I/O error: block 0x575b6d70
> ("xfs_trans_read_buf_map") error 4 numblks 16

#define EINTR            4      /* Interrupted system call */

That's an EINTR error coming from the block layer. This is most
definitely not an error that the block layer should be returning to
the filesystem.

IOWs, you've hit a bcache bug.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Stefan Priebe - Profihost AG]

Am 02.08.2017 um 00:23 schrieb Dave Chinner:
> On Tue, Aug 01, 2017 at 10:12:57AM +0200, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> under very high load i'm getting the following error on a bcache device
>> using XFS:
>> XFS (bcache1): metadata I/O error: block 0x575b6d70
>> ("xfs_trans_read_buf_map") error 4 numblks 16
> 
> #define EINTR            4      /* Interrupted system call */
> 
> That's an EINTR error coming from the block layer. This is most
> definitely not an error that the block layer should be returning to
> the filesystem.
> 
> IOWs, you've hit a bcache bug.

Yes, thanks!

Stefan

> Cheers,
> 
> Dave.
> 

metadata I/O error: block 0x575b6d70,("xfs_trans_read_buf_map") error 4 numblks 16

[Stefan Priebe - Profihost AG]

Hello,

under very high load i'm getting the following error on a bcache device
using XFS:
XFS (bcache1): metadata I/O error: block 0x575b6d70
("xfs_trans_read_buf_map") error 4 numblks 16
XFS (bcache1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -4.
XFS (bcache1): xfs_do_force_shutdown(0x8) called from line 3519 of file
fs/xfs/xfs_inode.c. Return address = 0xffffffff993441ff
XFS (bcache1): Corruption of in-memory data detected. Shutting down
filesystem
XFS (bcache1): Please umount the filesystem and rectify the problem(s)
XFS (bcache1): xfs_log_force: error -5 returned.
XFS (bcache1): xfs_log_force: error -5 returned.
XFS (bcache1): xfs_log_force: error -5 returned.

The used kernel is 4.4.79 based while using an opensuses patchset. Is
this something which is a known problem or something suse has broken?

Thanks!

Greets,
Stefan