Re: [PATCH v8 17/18] scsi: megaraid_sas: Added support for shared host tagset for cpuhotplug

[Ming Lei <ming.lei@redhat.com>]

On Wed, Nov 11, 2020 at 12:57:59PM +0530, Sumit Saxena wrote:
> On Tue, Nov 10, 2020 at 11:12 PM John Garry <john.garry@huawei.com> wrote:
> >
> > On 09/11/2020 14:05, John Garry wrote:
> > > On 09/11/2020 13:39, Qian Cai wrote:
> > >>> I suppose I could try do this myself also, but an authentic version
> > >>> would be nicer.
> > >> The closest one I have here is:
> > >> https://cailca.coding.net/public/linux/mm/git/files/master/arm64.config
> > >>
> > >> but it only selects the Thunder X2 platform and needs to manually select
> > >> CONFIG_MEGARAID_SAS=m to start with, but none of arm64 systems here have
> > >> megaraid_sas.
> > >
> > > Thanks, I'm confident I can fix it up to get it going on my Huawei arm64
> > > D06CS.
> > >
> > > So that board has a megaraid sas card. In addition, it also has hisi_sas
> > > HW, which is another storage controller which we enabled this same
> > > feature which is causing the problem.
> > >
> > > I'll report back when I can.
> >
> > So I had to hack that arm64 config a bit to get it booting:
> > https://github.com/hisilicon/kernel-dev/commits/private-topic-sas-5.10-megaraid-hang
> >
> > Boot is ok on my board without the megaraid sas card, but includes
> > hisi_sas HW (which enables the equivalent option which is exposing the
> > problem).
> >
> > But the board with the megaraid sas boots very slowly, specifically
> > around the megaraid sas probe:
> >
> > : ttyS0 at MMIO 0x3f00002f8 (irq = 17, base_baud = 115200) is a 16550A
> > [   50.023726][    T1] printk: console [ttyS0] enabled
> > [   50.412597][    T1] megasas: 07.714.04.00-rc1
> > [   50.436614][    T5] megaraid_sas 0000:08:00.0: FW now in Ready state
> > [   50.450079][    T5] megaraid_sas 0000:08:00.0: 63 bit DMA mask and 63
> > bit consistent mask
> > [   50.467811][    T5] megaraid_sas 0000:08:00.0: firmware supports msix
> >         : (128)
> > [   50.845995][    T5] megaraid_sas 0000:08:00.0: requested/available
> > msix 128/128
> > [   50.861476][    T5] megaraid_sas 0000:08:00.0: current msix/online
> > cpus      : (128/128)
> > [   50.877616][    T5] megaraid_sas 0000:08:00.0: RDPQ mode     : (enabled)
> > [   50.891018][    T5] megaraid_sas 0000:08:00.0: Current firmware
> > supports maximum commands: 4077       LDIO threshold: 0
> > [   51.262942][    T5] megaraid_sas 0000:08:00.0: Performance mode
> > :Latency (latency index = 1)
> > [   51.280749][    T5] megaraid_sas 0000:08:00.0: FW supports sync cache
> >         : Yes
> > [   51.295451][    T5] megaraid_sas 0000:08:00.0:
> > megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
> > [   51.387474][    T5] megaraid_sas 0000:08:00.0: FW provided
> > supportMaxExtLDs: 1       max_lds: 64
> > [   51.404931][    T5] megaraid_sas 0000:08:00.0: controller type
> > : MR(2048MB)
> > [   51.419616][    T5] megaraid_sas 0000:08:00.0: Online Controller
> > Reset(OCR)  : Enabled
> > [   51.436132][    T5] megaraid_sas 0000:08:00.0: Secure JBOD support
> > : Yes
> > [   51.450265][    T5] megaraid_sas 0000:08:00.0: NVMe passthru support
> > : Yes
> > [   51.464757][    T5] megaraid_sas 0000:08:00.0: FW provided TM
> > TaskAbort/Reset timeout        : 6 secs/60 secs
> > [   51.484379][    T5] megaraid_sas 0000:08:00.0: JBOD sequence map
> > support     : Yes
> > [   51.499607][    T5] megaraid_sas 0000:08:00.0: PCI Lane Margining
> > support    : No
> > [   51.547610][    T5] megaraid_sas 0000:08:00.0: NVME page size
> > : (4096)
> > [   51.608635][    T5] megaraid_sas 0000:08:00.0:
> > megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
> > [   51.630285][    T5] megaraid_sas 0000:08:00.0: INIT adapter done
> > [   51.649854][    T5] megaraid_sas 0000:08:00.0: pci id
> > : (0x1000)/(0x0016)/(0x19e5)/(0xd215)
> > [   51.667873][    T5] megaraid_sas 0000:08:00.0: unevenspan support    : no
> > [   51.681646][    T5] megaraid_sas 0000:08:00.0: firmware crash dump   : no
> > [   51.695596][    T5] megaraid_sas 0000:08:00.0: JBOD sequence map
> > : enabled
> > [   51.711521][    T5] megaraid_sas 0000:08:00.0: Max firmware commands:
> > 4076 shared with nr_hw_queues = 127
> > [   51.733056][    T5] scsi host0: Avago SAS based MegaRAID driver
> > [   65.304363][    T5] scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG
> > MZ7KH1T9 404Q PQ: 0 ANSI: 6
> > [   65.392401][    T5] scsi 0:0:1:0: Direct-Access     ATA      SAMSUNG
> > MZ7KH1T9 404Q PQ: 0 ANSI: 6
> > [   79.508307][    T5] scsi 0:0:65:0: Enclosure         HUAWEI
> > Expander 12Gx16  131  PQ: 0 ANSI: 6
> > [  183.965109][   C14] random: fast init done
> >
> > Notice the 14 and 104 second delays.
> >
> > But does boot fully to get to the console. I'll wait for further issues,
> > which you guys seem to experience after a while.
> >
> > Thanks,
> > John
> "megaraid_sas" driver calls “scsi_scan_host()” to discover SCSI
> devices. In this failure case, scsi_scan_host() is taking a long time
> to complete, hence causing delay in system boot.
> With "host_tagset" enabled, scsi_scan_host() takes around 20 mins.
> With "host_tagset" disabled, scsi_scan_host() takes upto 5-8 mins.
> 
> The scan time depends upon the number of scsi channels and devices per
> scsi channel is exposed by LLD.
> megaraid_sas driver exposes 4 channels and 128 drives per channel.
> 
> Each target scan takes 2 seconds (in case of failure with host_tagset
> enabled).  That's why driver load completes after ~20 minutes. See
> below:
> 
> [  299.725271] kobject: 'target18:0:96': free name
> [  301.681267] kobject: 'target18:0:97' (00000000987c7f11):
> kobject_cleanup, parent 0000000000000000
> [  301.681269] kobject: 'target18:0:97' (00000000987c7f11): calling
> ktype release
> [  301.681273] kobject: 'target18:0:97': free name
> [  303.575268] kobject: 'target18:0:98' (00000000a8c34149):
> kobject_cleanup, parent 0000000000000000
> 
> In Qian's kernel .config, async scsi scan is disabled so in failure
> case SCSI scan type is synchronous.
> Below is the stack trace when scsi_scan_host() hangs:
> 
> [<0>] __wait_rcu_gp+0x134/0x170
> [<0>] synchronize_rcu.part.80+0x53/0x60
> [<0>] blk_free_flush_queue+0x12/0x30

Can this issue disappear by applying the following change?

diff --git a/block/blk-flush.c b/block/blk-flush.c
index e32958f0b687..b1fe6176d77f 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -469,9 +469,6 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,
 	INIT_LIST_HEAD(&fq->flush_queue[1]);
 	INIT_LIST_HEAD(&fq->flush_data_in_flight);
 
-	lockdep_register_key(&fq->key);
-	lockdep_set_class(&fq->mq_flush_lock, &fq->key);
-
 	return fq;
 
  fail_rq:
@@ -486,7 +483,6 @@ void blk_free_flush_queue(struct blk_flush_queue *fq)
 	if (!fq)
 		return;
 
-	lockdep_unregister_key(&fq->key);
 	kfree(fq->flush_rq);
 	kfree(fq);
 }


Thanks, 
Ming