ServeRAID-BR10i (LSI SAS1068E) strageness

ServeRAID-BR10i (LSI SAS1068E) strageness

[Eyal Lebedinsky]

This is an IBM ServeRAID-BR10i. Identified at boot time as
	LSISAS1068E B3, FwRev=011b5600h
I was using another controller (HighPoint RocketRAID 2720SGL) which gives me grief
so decided to re-commission my old (slow) controller. It worked perfectly until retired
4 years ago.

I want to use this old controller, to confirm that my recent problems are from my controller,
and if so acquire a reliable one.

The LSI controller POST messages indicated it seed all 7 disks, and they are also
present in the linux messages, but the partitions are not there. I did not enter
the controller's configuration dialogue.

Each disk has one (unformatted, created with gparted) partition which is a member of
a RAID6.

The messages in the log are these:

Sep 22 14:59:18 e7 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 20, phy 0, sas_addr 0x70944e23a5c4b492
Sep 22 14:59:18 e7 kernel: scsi 6:0:0:0: Direct-Access     ATA      WDC WD4003FZEX-0 1A01 PQ: 0 ANSI: 5
Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: Attached scsi generic sg2 type 0
Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] 4294967294 512-byte logical blocks: (2.20 TB/2.00 TiB)
Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] Write Protect is off
Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Unlike the usual messages:

Sep 23 01:05:17 e7 kernel: ata7.00: ATA-8: WDC WD4001FAEX-00MJRA0, 01.01L01, max UDMA/133
Sep 23 01:05:17 e7 kernel: ata7.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
Sep 23 01:05:17 e7 kernel: ata7.00: configured for UDMA/133
Sep 23 01:05:17 e7 kernel: scsi 6:0:0:0: Direct-Access     ATA      WDC WD4001FAEX-0 1L01 PQ: 0 ANSI: 5
Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: Attached scsi generic sg2 type 0
Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] Write Protect is off
Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 23 01:05:19 e7 kernel: sdc: sdc1
Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] Attached SCSI disk

In short, no ata?.00 devices and no 'sdc: sdc1' partition list.

As an aside, I notice that the messages from parted are a bit different from what I have recorded.
My records are from when I was running f19 (now on f26) so maybe the utilities changed.

The old record (from f19):

$ sudo parted /dev/sdj print
Model: ATA WDC WD4001FAEX-0 (scsi)
Disk /dev/sdj: 4001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
  1      1049kB  4001GB  4001GB

What I see now (from f26):

$ sudo parted /dev/sdc print
Model: ATA WDC WD4001FAEX-0 (scsi)
Disk /dev/sdc: 4001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
  1      1049kB  4001GB  4001GB                     msftdata

The partition is now showing a flag 'msftdata'.
Is this a utility change (fdisk is also slightly different) or did the LSI touch the partition table?

TIA

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: ServeRAID-BR10i (LSI SAS1068E) strageness

[Roman Mamedov]

On Fri, 22 Sep 2017 18:11:36 +1000
Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:

> In short, no ata?.00 devices

This can be normal, one controller I use now is "LSI Logic / Symbios Logic SAS2008 PCI-Express
Fusion-MPT SAS-2" (mpt3sas driver) and it also does not use the ata devices at all.

$ ls /sys/block/sd* -la
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sda -> ../devices/pci0000:00/0000:00:11.0/ata1/host1/target1:0:0/1:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdb -> ../devices/pci0000:00/0000:00:11.0/ata2/host2/target2:0:0/2:0:0:0/block/sdb
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdc -> ../devices/pci0000:00/0000:00:11.0/ata3/host3/target3:0:0/3:0:0:0/block/sdc
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdd -> ../devices/pci0000:00/0000:00:11.0/ata4/host4/target4:0:0/4:0:0:0/block/sdd
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sde -> ../devices/pci0000:00/0000:00:11.0/ata5/host5/target5:0:0/5:0:0:0/block/sde
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdf -> ../devices/pci0000:00/0000:00:11.0/ata6/host6/target6:0:0/6:0:0:0/block/sdf
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdg -> ../devices/pci0000:00/0000:00:02.0/0000:01:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sdg
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdh -> ../devices/pci0000:00/0000:00:02.0/0000:01:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdh
lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdi -> ../devices/pci0000:00/0000:00:02.0/0000:01:00.0/host0/port-0:2/end_device-0:2/target0:0:2/0:0:2:0/block/sdi

3 disks on the SAS2008 controller.

> and no 'sdc: sdc1' partition list.

Did you check if there is the actual /dev/sdc1 present? Tried to access/mount/re-add it to RAID?

-- 
With respect,
Roman

Re: ServeRAID-BR10i (LSI SAS1068E) strageness

[Eyal Lebedinsky]

On 22/09/17 18:36, Roman Mamedov wrote:
> On Fri, 22 Sep 2017 18:11:36 +1000
> Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:
> 
>> In short, no ata?.00 devices
> 
> This can be normal, one controller I use now is "LSI Logic / Symbios Logic SAS2008 PCI-Express
> Fusion-MPT SAS-2" (mpt3sas driver) and it also does not use the ata devices at all.
> 
> $ ls /sys/block/sd* -la
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sda -> ../devices/pci0000:00/0000:00:11.0/ata1/host1/target1:0:0/1:0:0:0/block/sda
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdb -> ../devices/pci0000:00/0000:00:11.0/ata2/host2/target2:0:0/2:0:0:0/block/sdb
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdc -> ../devices/pci0000:00/0000:00:11.0/ata3/host3/target3:0:0/3:0:0:0/block/sdc
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdd -> ../devices/pci0000:00/0000:00:11.0/ata4/host4/target4:0:0/4:0:0:0/block/sdd
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sde -> ../devices/pci0000:00/0000:00:11.0/ata5/host5/target5:0:0/5:0:0:0/block/sde
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdf -> ../devices/pci0000:00/0000:00:11.0/ata6/host6/target6:0:0/6:0:0:0/block/sdf
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdg -> ../devices/pci0000:00/0000:00:02.0/0000:01:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sdg
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdh -> ../devices/pci0000:00/0000:00:02.0/0000:01:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdh
> lrwxrwxrwx 1 root root 0 Sep 22 13:28 /sys/block/sdi -> ../devices/pci0000:00/0000:00:02.0/0000:01:00.0/host0/port-0:2/end_device-0:2/target0:0:2/0:0:2:0/block/sdi
> 
> 3 disks on the SAS2008 controller.
> 
>> and no 'sdc: sdc1' partition list.
> 
> Did you check if there is the actual /dev/sdc1 present? Tried to access/mount/re-add it to RAID?

Thanks Roman,

Yes, there were no /dev/sd?1 devices, which are the RAID members.

I will install this card again and look around a bit more. I am waiting to hear
if the controller is likely to have touched the disks before testing again.

cheers

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Resolved? ServeRAID-BR10i (LSI SAS1068E) strageness

[Eyal Lebedinsky]

On 22/09/17 18:11, Eyal Lebedinsky wrote:
> This is an IBM ServeRAID-BR10i. Identified at boot time as
>      LSISAS1068E B3, FwRev=011b5600h
> I was using another controller (HighPoint RocketRAID 2720SGL) which gives me grief
> so decided to re-commission my old (slow) controller. It worked perfectly until retired
> 4 years ago.
> 
> I want to use this old controller, to confirm that my recent problems are from my controller,
> and if so acquire a reliable one.
> 
> The LSI controller POST messages indicated it seed all 7 disks, and they are also
> present in the linux messages, but the partitions are not there. I did not enter
> the controller's configuration dialogue.
> 
> Each disk has one (unformatted, created with gparted) partition which is a member of
> a RAID6.
> 
> The messages in the log are these:
> 
> Sep 22 14:59:18 e7 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 20, phy 0, sas_addr 0x70944e23a5c4b492
> Sep 22 14:59:18 e7 kernel: scsi 6:0:0:0: Direct-Access     ATA      WDC WD4003FZEX-0 1A01 PQ: 0 ANSI: 5
> Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: Attached scsi generic sg2 type 0
> Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] 4294967294 512-byte logical blocks: (2.20 TB/2.00 TiB)
> Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] Write Protect is off
> Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Re-reading the messages I now notice that the disks are identified as
	4294967294 512-byte logical blocks: (2.20 TB/2.00 TiB)
which is wrong (these are 4TB disks, as shown below).

I guess this controller, being a SATA-II, is not suitable for disks larger that 2TB?

Regards

> Unlike the usual messages:
> 
> Sep 23 01:05:17 e7 kernel: ata7.00: ATA-8: WDC WD4001FAEX-00MJRA0, 01.01L01, max UDMA/133
> Sep 23 01:05:17 e7 kernel: ata7.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
> Sep 23 01:05:17 e7 kernel: ata7.00: configured for UDMA/133
> Sep 23 01:05:17 e7 kernel: scsi 6:0:0:0: Direct-Access     ATA      WDC WD4001FAEX-0 1L01 PQ: 0 ANSI: 5
> Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: Attached scsi generic sg2 type 0
> Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
> Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] Write Protect is off
> Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Sep 23 01:05:19 e7 kernel: sdc: sdc1
> Sep 23 01:05:19 e7 kernel: sd 6:0:0:0: [sdc] Attached SCSI disk
> 
> In short, no ata?.00 devices and no 'sdc: sdc1' partition list.
> 
> As an aside, I notice that the messages from parted are a bit different from what I have recorded.
> My records are from when I was running f19 (now on f26) so maybe the utilities changed.
> 
> The old record (from f19):
> 
> $ sudo parted /dev/sdj print
> Model: ATA WDC WD4001FAEX-0 (scsi)
> Disk /dev/sdj: 4001GB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start   End     Size    File system  Name  Flags
>   1      1049kB  4001GB  4001GB
> 
> What I see now (from f26):
> 
> $ sudo parted /dev/sdc print
> Model: ATA WDC WD4001FAEX-0 (scsi)
> Disk /dev/sdc: 4001GB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start   End     Size    File system  Name  Flags
>   1      1049kB  4001GB  4001GB                     msftdata
> 
> The partition is now showing a flag 'msftdata'.
> Is this a utility change (fdisk is also slightly different) or did the LSI touch the partition table?
> 
> TIA
> 

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: Resolved? ServeRAID-BR10i (LSI SAS1068E) strageness

[Roman Mamedov]

On Thu, 28 Sep 2017 15:03:23 +1000
Eyal Lebedinsky <eyal@eyal.emu.id.au> wrote:

> > Sep 22 14:59:18 e7 kernel: mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 20, phy 0, sas_addr 0x70944e23a5c4b492
> > Sep 22 14:59:18 e7 kernel: scsi 6:0:0:0: Direct-Access     ATA      WDC WD4003FZEX-0 1A01 PQ: 0 ANSI: 5
> > Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: Attached scsi generic sg2 type 0
> > Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] 4294967294 512-byte logical blocks: (2.20 TB/2.00 TiB)
> > Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] Write Protect is off
> > Sep 22 14:59:18 e7 kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> 
> Re-reading the messages I now notice that the disks are identified as
> 	4294967294 512-byte logical blocks: (2.20 TB/2.00 TiB)
> which is wrong (these are 4TB disks, as shown below).
> 
> I guess this controller, being a SATA-II, is not suitable for disks larger that 2TB?

Great catch. These are not interdependent though, it's not 2TB incapable just
because of SATA2. There are SATA2 controllers supporting 2TB+ drives.

But come to think of it, I now remember the monumental article at [1]
warning about the 1068E, "does not support drives larger than 2TB."

[1]
http://blog.zorinaq.com/from-32-to-2-ports-ideal-satasas-controllers-for-zfs-linux-md-ra/

-- 
With respect,
Roman