Unable to add journal device to RAID 6 array

Unable to add journal device to RAID 6 array

[James Stephenson]

Hi,

First time using this (and indeed any) mailing list, so apologies if I
violate any etiquette I'm not aware of!

I'm trying to add a write-back journal device to an existing RAID 6
array, and it's proving difficult. Essentially I did this:

1. Put the array in read-only
2. Attempt to add a journal device to the array
3. md said no because the array has a bitmap
4. I tried to remove the bitmap, and it said no: 'md: couldn't update
array info. -16'
5. I rebooted
6. The array wouldn't start, and to my surprise was listed as having a
journal device. Here's what it looked like:
https://gist.github.com/jstephenson/1db2008c4243c2539d029f1f4706dc14
7. It was _very_ unhappy, and refused to do anything: 'md/raid:md126:
array cannot have both journal and bitmap'
8. The only thing I could do was to zero the superblock on the journal
device, and then fortunately everything assemble again nicely (with
bitmap still in place)

So, after a bit of messing around the array is back to where I
started—RAID 6 with internal bitmap. However, I still cannot remove
its bitmap.

1. sudo mdadm --grow /dev/md126 --bitmap=none
2. md: couldn't update array info. -16

I would greatly appreciate any thoughts on why this might be.

Kernel 5.10.106, mdadm 4.1

Thanks,

James

Re: Unable to add journal device to RAID 6 array

[Guoqing Jiang]

On 5/6/22 6:42 PM, James Stephenson wrote:
> Hi,
>
> First time using this (and indeed any) mailing list, so apologies if I
> violate any etiquette I'm not aware of!
>
> I'm trying to add a write-back journal device to an existing RAID 6
> array, and it's proving difficult. Essentially I did this:
>
> 1. Put the array in read-only
> 2. Attempt to add a journal device to the array
> 3. md said no because the array has a bitmap
> 4. I tried to remove the bitmap, and it said no: 'md: couldn't update
> array info. -16'
> 5. I rebooted
> 6. The array wouldn't start, and to my surprise was listed as having a
> journal device. Here's what it looked like:
> https://gist.github.com/jstephenson/1db2008c4243c2539d029f1f4706dc14
> 7. It was _very_ unhappy, and refused to do anything: 'md/raid:md126:
> array cannot have both journal and bitmap'
> 8. The only thing I could do was to zero the superblock on the journal
> device, and then fortunately everything assemble again nicely (with
> bitmap still in place)
>
> So, after a bit of messing around the array is back to where I
> started—RAID 6 with internal bitmap. However, I still cannot remove
> its bitmap.
>
> 1. sudo mdadm --grow /dev/md126 --bitmap=none
> 2. md: couldn't update array info. -16

-16 means EBUSY, so probably the array needs resync or recovery per

https://elixir.bootlin.com/linux/v5.10.106/source/drivers/md/md.c#L7374

Thanks,
Guoqing

Re: Unable to add journal device to RAID 6 array

[James Stephenson]

Thank you for your reply. I did check the source and found the same
block, however I’m not familiar enough to understand where I would see
this condition indicated?

As of now the array seems perfectly normal:
https://gist.github.com/jstephenson/0b615aab33bb8157a3876471ef50424e

James


On Fri, 6 May 2022 at 12:48, Guoqing Jiang <guoqing.jiang@linux.dev> wrote:
>
>
>
> On 5/6/22 6:42 PM, James Stephenson wrote:
> > Hi,
> >
> > First time using this (and indeed any) mailing list, so apologies if I
> > violate any etiquette I'm not aware of!
> >
> > I'm trying to add a write-back journal device to an existing RAID 6
> > array, and it's proving difficult. Essentially I did this:
> >
> > 1. Put the array in read-only
> > 2. Attempt to add a journal device to the array
> > 3. md said no because the array has a bitmap
> > 4. I tried to remove the bitmap, and it said no: 'md: couldn't update
> > array info. -16'
> > 5. I rebooted
> > 6. The array wouldn't start, and to my surprise was listed as having a
> > journal device. Here's what it looked like:
> > https://gist.github.com/jstephenson/1db2008c4243c2539d029f1f4706dc14
> > 7. It was _very_ unhappy, and refused to do anything: 'md/raid:md126:
> > array cannot have both journal and bitmap'
> > 8. The only thing I could do was to zero the superblock on the journal
> > device, and then fortunately everything assemble again nicely (with
> > bitmap still in place)
> >
> > So, after a bit of messing around the array is back to where I
> > started—RAID 6 with internal bitmap. However, I still cannot remove
> > its bitmap.
> >
> > 1. sudo mdadm --grow /dev/md126 --bitmap=none
> > 2. md: couldn't update array info. -16
>
> -16 means EBUSY, so probably the array needs resync or recovery per
>
> https://elixir.bootlin.com/linux/v5.10.106/source/drivers/md/md.c#L7374
>
> Thanks,
> Guoqing

Re: Unable to add journal device to RAID 6 array

[Guoqing Jiang]

On 5/6/22 7:55 PM, James Stephenson wrote:
> Thank you for your reply. I did check the source and found the same
> block, however I’m not familiar enough to understand where I would see
> this condition indicated?
>
> As of now the array seems perfectly normal:
> https://gist.github.com/jstephenson/0b615aab33bb8157a3876471ef50424e

Seems the bitmap is not clean, so resync thread might be running.

bitmap: 3/88 pages [12KB], 65536KB chunk

I suppose you can find relevant threads with "ps aux | grep md126", and also
the output of "cat /proc/mdstat" could be helpful.

Thanks,
Guoqing

Re: Unable to add journal device to RAID 6 array

[James Stephenson]

Thanks for your reply. The bitmap state there was because the array
was active at the time.

I've found the issue however: it turns out that a previous 'check'
must have been interrupted by power loss or similar, and so the array
still had memory of this in some way (perhaps I could have seen this
state in the sysfs tree somewhere?). After reading the comment in
https://elixir.bootlin.com/linux/v5.10.106/source/drivers/md/md.h#L348
I thought I'd try a 'check' followed by an 'idle', and indeed after
that I was able to successfully remove the bitmap and add a journal
device :)!

Thanks,

James

On Fri, 6 May 2022 at 13:05, Guoqing Jiang <guoqing.jiang@linux.dev> wrote:
>
>
>
> On 5/6/22 7:55 PM, James Stephenson wrote:
> > Thank you for your reply. I did check the source and found the same
> > block, however I’m not familiar enough to understand where I would see
> > this condition indicated?
> >
> > As of now the array seems perfectly normal:
> > https://gist.github.com/jstephenson/0b615aab33bb8157a3876471ef50424e
>
> Seems the bitmap is not clean, so resync thread might be running.
>
> bitmap: 3/88 pages [12KB], 65536KB chunk
>
> I suppose you can find relevant threads with "ps aux | grep md126", and also
> the output of "cat /proc/mdstat" could be helpful.
>
> Thanks,
> Guoqing