Re: bcache detach lead to xfs force shutdown

[zhangzhen.email@gmail.com]

On 3/4/22 4:42 PM, Coly Li <colyli@suse.de> wrote:
> On 3/4/22 4:22 PM, Zhang Zhen wrote:
> >
> > On 3/2/22 5:19 PM, Coly Li wrote:
> >> On 2/23/22 8:26 PM, Zhang Zhen wrote:
> >>>
> >>> 在 2022/2/23 下午5:03, Coly Li 写道:
> >>>> On 2/21/22 5:33 PM, Zhang Zhen wrote:
> >>>>> Hi coly，
> >>>>>
> >>>>> We encounted a bcache detach problem, during the io process，the 
> >>>>> cache device become missing.
> >>>>>
> >>>>> The io error status returned to xfs， and in some case， the xfs 
> >>>>> do force shutdown.
> >>>>>
> >>>>> The dmesg as follows:
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: 
> >>>>> IO error on writing btree.
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
> >>>>> Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in 
> >>>>> "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: 
> >>>>> error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
> >>>>> Feb  2 20:59:23  kernel: journal io error
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling 
> >>>>> caching
> >>>>> Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
> >>>>> Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() 
> >>>>> stop_when_cache_set_failed of bcache43 is "auto" and cache is 
> >>>>> clean, keep it alive.
> >>>>> Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in 
> >>>>> "xlog_iodone" at daddr 0x400123e60 len 64 error 12
> >>>>> Feb  2 20:59:23  kernel: XFS (bcache43): 
> >>>>> xfs_do_force_shutdown(0x2) called from line 1298 of file 
> >>>>> fs/xfs/xfs_log.c. Return address = 00000000c1c8077f
> >>>>> Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. 
> >>>>> Shutting down filesystem
> >>>>> Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the 
> >>>>> filesystem and rectify the problem(s)
> >>>>>
> >>>>>
> >>>>> We checked the code, the error status is returned in 
> >>>>> cached_dev_make_request and closure_bio_submit function.
> >>>>>
> >>>>> 1180 static blk_qc_t cached_dev_make_request(struct request_queue *q,
> >>>>> 1181                     struct bio *bio)
> >>>>> 1182 {
> >>>>> 1183     struct search *s;
> >>>>> 1184     struct bcache_device *d = bio->bi_disk->private_data;
> >>>>> 1185     struct cached_dev *dc = container_of(d, struct 
> >>>>> cached_dev, disk);
> >>>>> 1186     int rw = bio_data_dir(bio);
> >>>>> 1187
> >>>>> 1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, 
> >>>>> &d->c->flags)) ||
> >>>>> 1189              dc->io_disable)) {
> >>>>> 1190         bio->bi_status = BLK_STS_IOERR;
> >>>>> 1191         bio_endio(bio);
> >>>>> 1192         return BLK_QC_T_NONE;
> >>>>> 1193     }
> >>>>>
> >>>>>  901 static inline void closure_bio_submit(struct cache_set *c,
> >>>>>  902                       struct bio *bio,
> >>>>>  903                       struct closure *cl)
> >>>>>  904 {
> >>>>>  905     closure_get(cl);
> >>>>>  906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) {
> >>>>>  907         bio->bi_status = BLK_STS_IOERR;
> >>>>>  908         bio_endio(bio);
> >>>>>  909         return;
> >>>>>  910     }
> >>>>>  911     generic_make_request(bio);
> >>>>>  912 }
> >>>>>
> >>>>> Can the cache set detached and don't return error status to fs?
> >>>>
> >>>>
> >>>> Hi Zhang,
> >>>>
> >>>>
> >>>> What is your kernel version and where do you get the kernel?
> >>>> My kernel version is 4.18 of Centos.
> >>> The code of this part is same with upstream kernel.
> >>>> It seems like an as designed behavior, could you please describe 
> >>>> more detail about the operation sequence?
> >>>>
> >>> Yes, i think so too.
> >>> The reproduce opreation as follows:
> >>> 1. mount a bcache disk with xfs
> >>>
> >>> /dev/bcache1 on /media/disk1 type xfs
> >>>
> >>> 2. run ls in background
> >>> #!/bin/bash
> >>>
> >>> while true
> >>> do
> >>>   echo 2 > /proc/sys/vm/drop_caches
> >>>   ls -R /media/disk1 > /dev/null
> >>> done
> >>>
> >>>
> >>> 3. remove cache disk sdc
> >>> echo 1 >/sys/block/sdc/device/delete
> >>>
> >>> 4. dmesg should get xfs error
> >>>
> >>> I write a patch to improve，please help to review it, thanks.
> >
> >>
> >> Hold on. Why do you think it should be fixed? As I said, it is 
> >> as-designed behavior.
> >>
> > We use bcache in writearound mode, just cache read io.
> > Currently, bcache return io error during detach, randomly lead to
> > xfs force shutdown.
> >
> > After bcache auto detach finished, some dir read write normaly, but
> > the others can't read write because of xfs force shutdown.
> > This inconsistency confuses filesystem users.
> 
> 
> Hi Zhen and Nix,
> 
> OK, I come to realize the motivation. Yes you are right, this is an 
> awkward issue and good to be fixed.
> 
Hi Coly,

So you will pick this patch into your tree ？
> 
> Coly Li
> 
>