mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Gavin Guo]

There is a problem in the latest upstream kernel with the device:

$ grep -i lsi lspci
03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios
Logic SAS1068 PCI-X Fusion-MPT SAS [1000:0054] (rev 01)

The device is simulated by the VMware ESXi 5.5

When hotplugging a new disk to the Guest Ubuntu OS, the latest kernel
cannot automatically probe the disk. However, on the v3.19.0-80.88
kernel, the disk can be dynamically probed and show the following
info message:

mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
sas_addr 0x5000c29a6bdae0f5
scsi 2:0:1:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0
ANSI: 2
sd 2:0:1:0: Attached scsi generic sg2 type 0
sd 2:0:1:0: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
sd 2:0:1:0: [sdb] Write Protect is off
sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00
sd 2:0:1:0: [sdb] Cache data unavailable
sd 2:0:1:0: [sdb] Assuming drive cache: write through
 sdb: unknown partition table
sd 2:0:1:0: [sdb] Attached SCSI disk

After looking up the message:
mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
sas_addr 0x5000c29a6bdae0f5

I found it comes from the path:
mptsas_firmware_event_work -> mptsas_send_sas_event ->
mptsas_hotplug_work -> mptsas_add_end_device

I'll appreciate if anyone can give the idea: If it's possible that the
irq from the simulated LSI SAS controller didn't come in to trigger
the event? However, it can work on the v3.19 kernel so if there is
any driver implementation issue in the latest kernel.

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Hannes Reinecke]

On 09/27/2017 09:33 AM, Gavin Guo wrote:
> There is a problem in the latest upstream kernel with the device:
> 
> $ grep -i lsi lspci
> 03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios
> Logic SAS1068 PCI-X Fusion-MPT SAS [1000:0054] (rev 01)
> 
> The device is simulated by the VMware ESXi 5.5
> 
> When hotplugging a new disk to the Guest Ubuntu OS, the latest kernel
> cannot automatically probe the disk. However, on the v3.19.0-80.88
> kernel, the disk can be dynamically probed and show the following
> info message:
> 
> mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
> sas_addr 0x5000c29a6bdae0f5
> scsi 2:0:1:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0
> ANSI: 2
> sd 2:0:1:0: Attached scsi generic sg2 type 0
> sd 2:0:1:0: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
> sd 2:0:1:0: [sdb] Write Protect is off
> sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00
> sd 2:0:1:0: [sdb] Cache data unavailable
> sd 2:0:1:0: [sdb] Assuming drive cache: write through
>  sdb: unknown partition table
> sd 2:0:1:0: [sdb] Attached SCSI disk
> 
> After looking up the message:
> mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
> sas_addr 0x5000c29a6bdae0f5
> 
> I found it comes from the path:
> mptsas_firmware_event_work -> mptsas_send_sas_event ->
> mptsas_hotplug_work -> mptsas_add_end_device
> 
> I'll appreciate if anyone can give the idea: If it's possible that the
> irq from the simulated LSI SAS controller didn't come in to trigger
> the event? However, it can work on the v3.19 kernel so if there is
> any driver implementation issue in the latest kernel.
> 
This is an issue with the mptsas driver, who originally assumed that no
system will have direct-attached SAS devices.
VMWare chose to implement exactly that, so the hotplug detection logic
is flawed here.
I'll be sending a patch.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Gavin Guo]

On Mon, Oct 2, 2017 at 10:32 PM, Hannes Reinecke <hare@suse.de> wrote:
> On 09/27/2017 09:33 AM, Gavin Guo wrote:
>> There is a problem in the latest upstream kernel with the device:
>>
>> $ grep -i lsi lspci
>> 03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios
>> Logic SAS1068 PCI-X Fusion-MPT SAS [1000:0054] (rev 01)
>>
>> The device is simulated by the VMware ESXi 5.5
>>
>> When hotplugging a new disk to the Guest Ubuntu OS, the latest kernel
>> cannot automatically probe the disk. However, on the v3.19.0-80.88
>> kernel, the disk can be dynamically probed and show the following
>> info message:
>>
>> mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
>> sas_addr 0x5000c29a6bdae0f5
>> scsi 2:0:1:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0
>> ANSI: 2
>> sd 2:0:1:0: Attached scsi generic sg2 type 0
>> sd 2:0:1:0: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
>> sd 2:0:1:0: [sdb] Write Protect is off
>> sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00
>> sd 2:0:1:0: [sdb] Cache data unavailable
>> sd 2:0:1:0: [sdb] Assuming drive cache: write through
>>  sdb: unknown partition table
>> sd 2:0:1:0: [sdb] Attached SCSI disk
>>
>> After looking up the message:
>> mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
>> sas_addr 0x5000c29a6bdae0f5
>>
>> I found it comes from the path:
>> mptsas_firmware_event_work -> mptsas_send_sas_event ->
>> mptsas_hotplug_work -> mptsas_add_end_device
>>
>> I'll appreciate if anyone can give the idea: If it's possible that the
>> irq from the simulated LSI SAS controller didn't come in to trigger
>> the event? However, it can work on the v3.19 kernel so if there is
>> any driver implementation issue in the latest kernel.
>>
> This is an issue with the mptsas driver, who originally assumed that no
> system will have direct-attached SAS devices.
> VMWare chose to implement exactly that, so the hotplug detection logic
> is flawed here.
> I'll be sending a patch.
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                Teamlead Storage & Networking
> hare@suse.de                                   +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)

Thank you in advance. Please add me to the CC list when you send out
the patch and I can help to verify that.

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Gavin Guo]

Hi Hannes,

Thank you for looking into the issue. If there is anything I can help
to test the patch? I appreciate your help. Thank you.

On Fri, Oct 6, 2017 at 9:08 AM, Gavin Guo <gavin.guo@canonical.com> wrote:
> On Mon, Oct 2, 2017 at 10:32 PM, Hannes Reinecke <hare@suse.de> wrote:
>> On 09/27/2017 09:33 AM, Gavin Guo wrote:
>>> There is a problem in the latest upstream kernel with the device:
>>>
>>> $ grep -i lsi lspci
>>> 03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios
>>> Logic SAS1068 PCI-X Fusion-MPT SAS [1000:0054] (rev 01)
>>>
>>> The device is simulated by the VMware ESXi 5.5
>>>
>>> When hotplugging a new disk to the Guest Ubuntu OS, the latest kernel
>>> cannot automatically probe the disk. However, on the v3.19.0-80.88
>>> kernel, the disk can be dynamically probed and show the following
>>> info message:
>>>
>>> mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
>>> sas_addr 0x5000c29a6bdae0f5
>>> scsi 2:0:1:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0
>>> ANSI: 2
>>> sd 2:0:1:0: Attached scsi generic sg2 type 0
>>> sd 2:0:1:0: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
>>> sd 2:0:1:0: [sdb] Write Protect is off
>>> sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00
>>> sd 2:0:1:0: [sdb] Cache data unavailable
>>> sd 2:0:1:0: [sdb] Assuming drive cache: write through
>>>  sdb: unknown partition table
>>> sd 2:0:1:0: [sdb] Attached SCSI disk
>>>
>>> After looking up the message:
>>> mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 1,
>>> sas_addr 0x5000c29a6bdae0f5
>>>
>>> I found it comes from the path:
>>> mptsas_firmware_event_work -> mptsas_send_sas_event ->
>>> mptsas_hotplug_work -> mptsas_add_end_device
>>>
>>> I'll appreciate if anyone can give the idea: If it's possible that the
>>> irq from the simulated LSI SAS controller didn't come in to trigger
>>> the event? However, it can work on the v3.19 kernel so if there is
>>> any driver implementation issue in the latest kernel.
>>>
>> This is an issue with the mptsas driver, who originally assumed that no
>> system will have direct-attached SAS devices.
>> VMWare chose to implement exactly that, so the hotplug detection logic
>> is flawed here.
>> I'll be sending a patch.
>>
>> Cheers,
>>
>> Hannes
>> --
>> Dr. Hannes Reinecke                Teamlead Storage & Networking
>> hare@suse.de                                   +49 911 74053 688
>> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
>> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
>> HRB 21284 (AG Nürnberg)
>
> Thank you in advance. Please add me to the CC list when you send out
> the patch and I can help to verify that.

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Hannes Reinecke]

On 10/27/2017 04:02 PM, Gavin Guo wrote:
> Hi Hannes,
> 
> Thank you for looking into the issue. If there is anything I can help
> to test the patch? I appreciate your help. Thank you.
> 
If you had checked linux-scsi you would have found this patch:
'[PATCH] mptsas: Fixup device hotplug for VMWare ESXi', which I guess is
already scheduled for inclusion in 4.14.
Anything else I could help you with?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Gavin Guo]

On Fri, Oct 27, 2017 at 10:53 PM, Hannes Reinecke <hare@suse.de> wrote:
> On 10/27/2017 04:02 PM, Gavin Guo wrote:
>> Hi Hannes,
>>
>> Thank you for looking into the issue. If there is anything I can help
>> to test the patch? I appreciate your help. Thank you.
>>
> If you had checked linux-scsi you would have found this patch:
> '[PATCH] mptsas: Fixup device hotplug for VMWare ESXi', which I guess is
> already scheduled for inclusion in 4.14.
> Anything else I could help you with?
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                Teamlead Storage & Networking
> hare@suse.de                                   +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)

Really appreciate your help. I will proceed the testing and keep you posted.

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Gavin Guo]

On Sat, Oct 28, 2017 at 11:35 AM, Gavin Guo <gavin.guo@canonical.com> wrote:
> On Fri, Oct 27, 2017 at 10:53 PM, Hannes Reinecke <hare@suse.de> wrote:
>> On 10/27/2017 04:02 PM, Gavin Guo wrote:
>>> Hi Hannes,
>>>
>>> Thank you for looking into the issue. If there is anything I can help
>>> to test the patch? I appreciate your help. Thank you.
>>>
>> If you had checked linux-scsi you would have found this patch:
>> '[PATCH] mptsas: Fixup device hotplug for VMWare ESXi', which I guess is
>> already scheduled for inclusion in 4.14.
>> Anything else I could help you with?
>>
>> Cheers,
>>
>> Hannes
>> --
>> Dr. Hannes Reinecke                Teamlead Storage & Networking
>> hare@suse.de                                   +49 911 74053 688
>> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
>> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
>> HRB 21284 (AG Nürnberg)
>
> Really appreciate your help. I will proceed the testing and keep you posted.

Hello Hannes,

I've tested the MPT SAS device controller, the patch works perfectly.
However, the MPT SPI still cannot work with the hotplugging. The lspci
of the LSI SPI device is listed:

00:10.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic
53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI [1000:0030] (rev 01)
        Subsystem: VMware LSI Logic Parallel SCSI Controller [15ad:1976]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64 (1500ns min, 63750ns max)
        Interrupt: pin A routed to IRQ 17
        Region 0: I/O ports at 1400 [size=256]
        Region 1: Memory at feba0000 (64-bit, non-prefetchable) [size=128K]
        Region 3: Memory at febc0000 (64-bit, non-prefetchable) [size=128K]
        [virtual] Expansion ROM at 40008000 [disabled] [size=16K]
        Kernel driver in use: mptspi

The vendor id/device id[1000:0030] can also be found in the
mptspi_pci_table(drivers/message/fusion/mptspi.c).

PCI_VENDOR_ID_LSI_LOGIC, MPI_MANUFACTPAGE_DEVID_53C1030

If there is anything I missed in the testing? I appreciate your help.

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Hannes Reinecke]

On 11/03/2017 04:38 AM, Gavin Guo wrote:
> On Sat, Oct 28, 2017 at 11:35 AM, Gavin Guo <gavin.guo@canonical.com> wrote:
>> On Fri, Oct 27, 2017 at 10:53 PM, Hannes Reinecke <hare@suse.de> wrote:
>>> On 10/27/2017 04:02 PM, Gavin Guo wrote:
>>>> Hi Hannes,
>>>>
>>>> Thank you for looking into the issue. If there is anything I can help
>>>> to test the patch? I appreciate your help. Thank you.
>>>>
>>> If you had checked linux-scsi you would have found this patch:
>>> '[PATCH] mptsas: Fixup device hotplug for VMWare ESXi', which I guess is
>>> already scheduled for inclusion in 4.14.
>>> Anything else I could help you with?
>>>
>>> Cheers,
>>>
>>> Hannes
>>> --
>>> Dr. Hannes Reinecke                Teamlead Storage & Networking
>>> hare@suse.de                                   +49 911 74053 688
>>> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
>>> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
>>> HRB 21284 (AG Nürnberg)
>>
>> Really appreciate your help. I will proceed the testing and keep you posted.
> 
> Hello Hannes,
> 
> I've tested the MPT SAS device controller, the patch works perfectly.
> However, the MPT SPI still cannot work with the hotplugging. The lspci
> of the LSI SPI device is listed:
> 
> 00:10.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic
> 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI [1000:0030] (rev 01)
>         Subsystem: VMware LSI Logic Parallel SCSI Controller [15ad:1976]
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>> TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 64 (1500ns min, 63750ns max)
>         Interrupt: pin A routed to IRQ 17
>         Region 0: I/O ports at 1400 [size=256]
>         Region 1: Memory at feba0000 (64-bit, non-prefetchable) [size=128K]
>         Region 3: Memory at febc0000 (64-bit, non-prefetchable) [size=128K]
>         [virtual] Expansion ROM at 40008000 [disabled] [size=16K]
>         Kernel driver in use: mptspi
> 
> The vendor id/device id[1000:0030] can also be found in the
> mptspi_pci_table(drivers/message/fusion/mptspi.c).
> 
> PCI_VENDOR_ID_LSI_LOGIC, MPI_MANUFACTPAGE_DEVID_53C1030
> 
> If there is anything I missed in the testing? I appreciate your help.
> 
Don't. Ever. Try.

SCSI Parallel hotplugging is so loosely defined that essentially every
vendor can (and did!) implement their own thing, and _still_ be spec
compliant.
Plus there is no guarantee that we even see events when a hot-add
occurs. So any thingie we put in here has a really high chance of
breaking other installations.
I wouldn't even bother to attempt this; it's virtualized anyway so you
can as well switch to the SAS emulation here.
Or use a different SCSI parallel emulation like the Symbios/LSI 53c800 one.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: mptsas driver cannot detect hotplugging disk with the LSI SCSI SAS1068 controller in Ubuntu guest on VMware

[Gavin Guo]

On Fri, Nov 3, 2017 at 6:59 PM, Hannes Reinecke <hare@suse.de> wrote:
> On 11/03/2017 04:38 AM, Gavin Guo wrote:
>> On Sat, Oct 28, 2017 at 11:35 AM, Gavin Guo <gavin.guo@canonical.com> wrote:
>>> On Fri, Oct 27, 2017 at 10:53 PM, Hannes Reinecke <hare@suse.de> wrote:
>>>> On 10/27/2017 04:02 PM, Gavin Guo wrote:
>>>>> Hi Hannes,
>>>>>
>>>>> Thank you for looking into the issue. If there is anything I can help
>>>>> to test the patch? I appreciate your help. Thank you.
>>>>>
>>>> If you had checked linux-scsi you would have found this patch:
>>>> '[PATCH] mptsas: Fixup device hotplug for VMWare ESXi', which I guess is
>>>> already scheduled for inclusion in 4.14.
>>>> Anything else I could help you with?
>>>>
>>>> Cheers,
>>>>
>>>> Hannes
>>>> --
>>>> Dr. Hannes Reinecke                Teamlead Storage & Networking
>>>> hare@suse.de                                   +49 911 74053 688
>>>> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
>>>> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
>>>> HRB 21284 (AG Nürnberg)
>>>
>>> Really appreciate your help. I will proceed the testing and keep you posted.
>>
>> Hello Hannes,
>>
>> I've tested the MPT SAS device controller, the patch works perfectly.
>> However, the MPT SPI still cannot work with the hotplugging. The lspci
>> of the LSI SPI device is listed:
>>
>> 00:10.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic
>> 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI [1000:0030] (rev 01)
>>         Subsystem: VMware LSI Logic Parallel SCSI Controller [15ad:1976]
>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B- DisINTx-
>>         Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>>> TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>         Latency: 64 (1500ns min, 63750ns max)
>>         Interrupt: pin A routed to IRQ 17
>>         Region 0: I/O ports at 1400 [size=256]
>>         Region 1: Memory at feba0000 (64-bit, non-prefetchable) [size=128K]
>>         Region 3: Memory at febc0000 (64-bit, non-prefetchable) [size=128K]
>>         [virtual] Expansion ROM at 40008000 [disabled] [size=16K]
>>         Kernel driver in use: mptspi
>>
>> The vendor id/device id[1000:0030] can also be found in the
>> mptspi_pci_table(drivers/message/fusion/mptspi.c).
>>
>> PCI_VENDOR_ID_LSI_LOGIC, MPI_MANUFACTPAGE_DEVID_53C1030
>>
>> If there is anything I missed in the testing? I appreciate your help.
>>
> Don't. Ever. Try.
>
> SCSI Parallel hotplugging is so loosely defined that essentially every
> vendor can (and did!) implement their own thing, and _still_ be spec
> compliant.
> Plus there is no guarantee that we even see events when a hot-add
> occurs. So any thingie we put in here has a really high chance of
> breaking other installations.
> I wouldn't even bother to attempt this; it's virtualized anyway so you
> can as well switch to the SAS emulation here.
> Or use a different SCSI parallel emulation like the Symbios/LSI 53c800 one.
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                Teamlead Storage & Networking
> hare@suse.de                                   +49 911 74053 688
> SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
> HRB 21284 (AG Nürnberg)

Hello Hannes,

I totally understood your concern and appreciate your help again on
the hotplugging patch for MPT SAS controller. :)