bblog overlap internal bitmap?

bblog overlap internal bitmap?

[Stan Liao]

Hi,
A md-raid (level 1) is created with 5 nvme drives and the metadata version is specified as 1.2. The following command is used.
sudo mdadm --create /dev/md0 --level=1 --raid-devices=5 /dev/nvme{1,2,3,4}n1 /dev/nvme4n2 --metadata=1.2
The capacities of nvme{1,2,3,4}n1 and nvme4n2 are 3.2TB, 1.92TB, 3.2TB, 512GB, and 512GB.
OS: 20.04.1-Ubuntu
mdadm version: v4.1 - 2018-10-01
After creation, we found that the size of the bitmap_super_t and internal bitmap is 16KB (this is concluded by observing FF value is filled from around byte offset 0x100 of LBA 0x10 to byte offset 0x1FF of LBA 0x1F), but the mdp_superblock_1.bblog_offset value is 0x10. As a result, the mdp_superblock_1 occupies LBA 0x08 ~ 0x0F; bitmap_super_t and internal bitmap occupy LBA 0x10 ~ 0x20; bblog occupies LBA 0x18 ~ 0x20.
If bblog and bitmap do overlap, I would like to know the size value used to calculate bitmap size and bblog_offset. The size value used to calculate bitmap size and bblog_offset is mdp_superblock_1.size or mdp_superblock_1.data_size? Thanks a lot.

Re: bblog overlap internal bitmap?

[Xiao Ni]

On Thu, Sep 21, 2023 at 11:22 AM Stan Liao <StanL@supermicro.com> wrote:
>
> Hi,
> A md-raid (level 1) is created with 5 nvme drives and the metadata version is specified as 1.2. The following command is used.
> sudo mdadm --create /dev/md0 --level=1 --raid-devices=5 /dev/nvme{1,2,3,4}n1 /dev/nvme4n2 --metadata=1.2
> The capacities of nvme{1,2,3,4}n1 and nvme4n2 are 3.2TB, 1.92TB, 3.2TB, 512GB, and 512GB.
> OS: 20.04.1-Ubuntu
> mdadm version: v4.1 - 2018-10-01
> After creation, we found that the size of the bitmap_super_t and internal bitmap is 16KB (this is concluded by observing FF value is filled from around byte offset 0x100 of LBA 0x10 to byte offset 0x1FF of LBA 0x1F), but the mdp_superblock_1.bblog_offset value is 0x10. As a result, the mdp_superblock_1 occupies LBA 0x08 ~ 0x0F; bitmap_super_t and internal bitmap occupy LBA 0x10 ~ 0x20; bblog occupies LBA 0x18 ~ 0x20.
> If bblog and bitmap do overlap, I would like to know the size value used to calculate bitmap size and bblog_offset. The size value used to calculate bitmap size and bblog_offset is mdp_superblock_1.size or mdp_superblock_1.data_size? Thanks a lot.
>

Hi

For super1.2 the layout should be:
|    4KB    |    4KB    |    bitmap space    |    head room    |   data |

The first 4KB is empty from the beginning of the disk. The second 4KB
is for md superblock. Then is bitmap suerblock. So if you want to see
the content of bitmap_super_t, the offset should be 0x2000?

Regards
Xiao

RE: bblog overlap internal bitmap?

[Stan Liao]

Xiao Ni wrote:
> On Thu, Sep 21, 2023 at 11:22 AM Stan Liao <StanL@supermicro.com> wrote:
> >
> > Hi,
> > A md-raid (level 1) is created with 5 nvme drives and the metadata version
> is specified as 1.2. The following command is used.
> > sudo mdadm --create /dev/md0 --level=1 --raid-devices=5
> > /dev/nvme{1,2,3,4}n1 /dev/nvme4n2 --metadata=1.2 The capacities of
> nvme{1,2,3,4}n1 and nvme4n2 are 3.2TB, 1.92TB, 3.2TB, 512GB, and 512GB.
> > OS: 20.04.1-Ubuntu
> > mdadm version: v4.1 - 2018-10-01
> > After creation, we found that the size of the bitmap_super_t and internal
> bitmap is 16KB (this is concluded by observing FF value is filled from around
> byte offset 0x100 of LBA 0x10 to byte offset 0x1FF of LBA 0x1F), but the
> mdp_superblock_1.bblog_offset value is 0x10. As a result, the
> mdp_superblock_1 occupies LBA 0x08 ~ 0x0F; bitmap_super_t and internal
> bitmap occupy LBA 0x10 ~ 0x20; bblog occupies LBA 0x18 ~ 0x20.
> > If bblog and bitmap do overlap, I would like to know the size value used to
> calculate bitmap size and bblog_offset. The size value used to calculate
> bitmap size and bblog_offset is mdp_superblock_1.size or
> mdp_superblock_1.data_size? Thanks a lot.
> >
> 
> Hi
> 
> For super1.2 the layout should be:
> |    4KB    |    4KB    |    bitmap space    |    head room    |
> data |
> 
> The first 4KB is empty from the beginning of the disk. The second 4KB is for
> md superblock. Then is bitmap suerblock. So if you want to see the content
> of bitmap_super_t, the offset should be 0x2000?
> 
> Regards
> Xiao

Right. The bitmap_super_t is at offset 0x2000 (absolute LBA address 0x10).
And mdadm write bitmap_super_t and bitmap from LBA 0x10 to LBA 0x1F.
Including bblog, super1.2 layout would be:
|  4KB  |  4KB  |  bitmap space  |  bblog  |  head room  | data |
We observe the bblog_offset value, which resides in mdp_superblock_1, is 0x10.
So we think there is an overlap between bitmap and bblog.

Re: bblog overlap internal bitmap?

[Xiao Ni]

On Fri, Sep 22, 2023 at 10:36 AM Stan Liao <StanL@supermicro.com> wrote:
>
> Xiao Ni wrote:
> > On Thu, Sep 21, 2023 at 11:22 AM Stan Liao <StanL@supermicro.com> wrote:
> > >
> > > Hi,
> > > A md-raid (level 1) is created with 5 nvme drives and the metadata version
> > is specified as 1.2. The following command is used.
> > > sudo mdadm --create /dev/md0 --level=1 --raid-devices=5
> > > /dev/nvme{1,2,3,4}n1 /dev/nvme4n2 --metadata=1.2 The capacities of
> > nvme{1,2,3,4}n1 and nvme4n2 are 3.2TB, 1.92TB, 3.2TB, 512GB, and 512GB.
> > > OS: 20.04.1-Ubuntu
> > > mdadm version: v4.1 - 2018-10-01
> > > After creation, we found that the size of the bitmap_super_t and internal
> > bitmap is 16KB (this is concluded by observing FF value is filled from around
> > byte offset 0x100 of LBA 0x10 to byte offset 0x1FF of LBA 0x1F), but the
> > mdp_superblock_1.bblog_offset value is 0x10. As a result, the
> > mdp_superblock_1 occupies LBA 0x08 ~ 0x0F; bitmap_super_t and internal
> > bitmap occupy LBA 0x10 ~ 0x20; bblog occupies LBA 0x18 ~ 0x20.
> > > If bblog and bitmap do overlap, I would like to know the size value used to
> > calculate bitmap size and bblog_offset. The size value used to calculate
> > bitmap size and bblog_offset is mdp_superblock_1.size or
> > mdp_superblock_1.data_size? Thanks a lot.
> > >
> >
> > Hi
> >
> > For super1.2 the layout should be:
> > |    4KB    |    4KB    |    bitmap space    |    head room    |
> > data |
> >
> > The first 4KB is empty from the beginning of the disk. The second 4KB is for
> > md superblock. Then is bitmap suerblock. So if you want to see the content
> > of bitmap_super_t, the offset should be 0x2000?
> >
> > Regards
> > Xiao
>
> Right. The bitmap_super_t is at offset 0x2000 (absolute LBA address 0x10).
> And mdadm write bitmap_super_t and bitmap from LBA 0x10 to LBA 0x1F.
> Including bblog, super1.2 layout would be:
> |  4KB  |  4KB  |  bitmap space  |  bblog  |  head room  | data |
> We observe the bblog_offset value, which resides in mdp_superblock_1, is 0x10.
> So we think there is an overlap between bitmap and bblog.
>

Because the bblog_offset is based on the start of superblock. So if
you want to calculate the offset based on the start of the disk, it
needs to add 4KB on bblog_offset. Now your bblog_offset is 0x10. It
means from the start of the superblock, it's 4K(superblock) +
4K(bitmap space). So your raid only has one page, right?

You can also you mdadm -E $member to check the "Bad Block Log"

Regards
Xiao