Re: [PATCH 07/10] hpsa: hide logical drives with format in progress from linux

[Tomas Henzl <thenzl@redhat.com>]

On 09/27/2013 04:41 PM, scameron@beardog.cce.hp.com wrote:
> On Fri, Sep 27, 2013 at 04:01:30PM +0200, Tomas Henzl wrote:
>> On 09/27/2013 03:34 PM, scameron@beardog.cce.hp.com wrote:
>>> On Fri, Sep 27, 2013 at 03:22:19PM +0200, Tomas Henzl wrote:
>>>> On 09/23/2013 08:34 PM, Stephen M. Cameron wrote:
>>>>> From: Stephen M. Cameron <scameron@beardog.cce.hp.com>
>>>>>
>>>>> SCSI mid layer doesn't seem to handle logical drives undergoing format
>>>>> very well.  scsi_add_device on such devices seems to result in hitting
>>>>> those devices with a TUR at a rate of 3Hz for awhile, transitioning
>>>>> to hitting them with a READ(10) at a much higher rate indefinitely,
>>>>> and at boot time, this prevents the system from coming up.  If we
>>>>> do not expose such devices to the kernel, it isn't bothered by them.
>>>> Is the result of this patch that the drive is no more visible for the user
>>>> and he can't follow the formatting progress? 
>>> Yes (subsequent patch monitors the progress and brings the drive
>>> online when it's ready).
>>>
>>>> I think a better option is to fix the kernel to handle formatting devices better
>>> Yeah, you're probably right. (This is what comes of writing code for all
>>> the distros then forward porting to kernel.org code.  Grumble-grumble-management
>>> grumble-grumble real-world problems.)
>>>
>>>> or harden the hpsa so it can cope with TURs or reads (ignore) from a formatting
>>>> device.
>>> I don't think hpsa driver had any problem with the TURs or READs though,
>>> they would be returned to the mid layer just fine (TUR returned sense data
>>> indicating not ready, format in progress, I forget what the reads
>>> returned, whatever the firmware filled in for the sense data, which
>>> was reasonable), but the mid-layer was relentless and just never
>>> really proceeded, iirc.
>>>
>>> Since we were trying to make this work on existing OSes where fixing the
>>> SCSI mid layer wasn't an option, we came up with this.
>> I'm actually glad that you care about existing OSes :)
> And the pain of porting would be much the same regardless of
> whether the port is forward or backward, I suppose.
>
>> Do you know whether the midlayer has similar problems with other drivers?
> No, not sure.   One thing that's a bit unusual about hpsa is it uses
> the scan_start and scan_finished members of scsi_host_template, so hpsa
> does its own scanning, rather than let the midlayer do the scanning which
> is due to Smart Array's weirdness around the vicinity of SCSI_REPORT_LUNS.
>
> I suspect that a lld driver calling scsi_add_device() on something which
> is NOT READY/FORMAT IN PROGRESS is what provokes the trouble.  Most drivers
> do not call scsi_add_device() directly at all, so it's quite possible most
> drivers do not experience such a problem. A few do call scsi_add_device()
> directly, like ipr or pmcraid, so these might conceivably have a similar
> problem.  
>
> We ran into this problem with what we call "Rapid Parity Initialization", which
> is what you get when the RAID controller leaves the logical volume in a NOT
> READY/FORMAT IN PROGRESS state and devotes itself entirely to initializing
> parity data and when that's done, then the volume starts acting normally.  
>
> Initializing the parity data can take quite a long time (hours), but not as
> long as initializing it on the fly under load, which, with very large,
> relatively slow drives can take nigh on forever, hence the "rapid" parity
> initialization moniker.  So, if those other RAID controllers don't have a
> similar feature that produces a relatively long lived NOT READY/FORMAT IN
> PROGRESS state, they may not bump into the problem.
>
> It has been awhile since I've tried letting the driver call scsi_add_device()
> on a device which is undergoing Rapid Parity Initialization, so I need to try
> that with current code and see how it behaves.  I haven't thought about how to
> fix it within the SCSI mid layer (presuming it still doesn't behave well)
> since previously we only concerned ourselves with avoiding provoking the
> undesirable behavior.
>
> -- steve

Thanks for the explanation. I hope I can look into this later. Sometimes later. When my
real-world problems go away...

>
>> Tomas
>>
>>>> Also maybe a cmd_special_free is missing - see below
>>> D'oh.  Ok, now that's just embarassing.  Thanks.
>>>
>>> -- steve
>>>
>>>> Cheers, Tomas
>>>> Signed-off-by: Stephen M. Cameron <scameron@beardog.cce.hp.com>
>>>> ---
>>>>  drivers/scsi/hpsa.c |   50 ++++++++++++++++++++++++++++++++++++++++++++++++--
>>>>  drivers/scsi/hpsa.h |    1 +
>>>>  2 files changed, 49 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c
>>>> index b7f405f..38e3af4 100644
>>>> --- a/drivers/scsi/hpsa.c
>>>> +++ b/drivers/scsi/hpsa.c
>>>> @@ -1010,6 +1010,20 @@ static void adjust_hpsa_scsi_table(struct ctlr_info *h, int hostno,
>>>>  	for (i = 0; i < nsds; i++) {
>>>>  		if (!sd[i]) /* if already added above. */
>>>>  			continue;
>>>> +
>>>> +		/* Don't add devices which are NOT READY, FORMAT IN PROGRESS
>>>> +		 * as the SCSI mid-layer does not handle such devices well.
>>>> +		 * It relentlessly loops sending TUR at 3Hz, then READ(10)
>>>> +		 * at 160Hz, and prevents the system from coming up.
>>>> +		 */
>>>> +		if (sd[i]->format_in_progress) {
>>>> +			dev_info(&h->pdev->dev,
>>>> +				"Logical drive format in progress, device c%db%dt%dl%d offline.\n",
>>>> +				h->scsi_host->host_no,
>>>> +				sd[i]->bus, sd[i]->target, sd[i]->lun);
>>>> +			continue;
>>>> +		}
>>>> +
>>>>  		device_change = hpsa_scsi_find_entry(sd[i], h->dev,
>>>>  					h->ndevices, &entry);
>>>>  		if (device_change == DEVICE_NOT_FOUND) {
>>>> @@ -1715,6 +1729,34 @@ static inline void hpsa_set_bus_target_lun(struct hpsa_scsi_dev_t *device,
>>>>  	device->lun = lun;
>>>>  }
>>>>  
>>>> +static unsigned char hpsa_format_in_progress(struct ctlr_info *h,
>>>> +		unsigned char scsi3addr[])
>>>> +{
>>>> +	struct CommandList *c;
>>>> +	unsigned char *sense, sense_key, asc, ascq;
>>>> +#define ASC_LUN_NOT_READY 0x04
>>>> +#define ASCQ_LUN_NOT_READY_FORMAT_IN_PROGRESS 0x04
>>>> +
>>>> +
>>>> +	c = cmd_special_alloc(h);
>>>> +	if (!c)
>>>> +		return 0;
>>>> +	fill_cmd(c, TEST_UNIT_READY, h, NULL, 0, 0, scsi3addr, TYPE_CMD);
>>>> +	hpsa_scsi_do_simple_cmd_core(h, c);
>>>> +	sense = c->err_info->SenseInfo;
>>>> +	sense_key = sense[2];
>>>> +	asc = sense[12];
>>>> +	ascq = sense[13];
>>>> +	if (c->err_info->CommandStatus == CMD_TARGET_STATUS &&
>>>> +		c->err_info->ScsiStatus == SAM_STAT_CHECK_CONDITION &&
>>>> +		sense_key == NOT_READY &&
>>>> +		asc == ASC_LUN_NOT_READY &&
>>>> +		ascq == ASCQ_LUN_NOT_READY_FORMAT_IN_PROGRESS)
>>>> +		return 1;
>>>> return^ without cmd_special_free
>>>>
>>>> +	cmd_special_free(h, c);
>>>> +	return 0;
>>>> +}
>>>> +
>>>>  static int hpsa_update_device_info(struct ctlr_info *h,
>>>>  	unsigned char scsi3addr[], struct hpsa_scsi_dev_t *this_device,
>>>>  	unsigned char *is_OBDR_device)
>>>> @@ -1753,10 +1795,14 @@ static int hpsa_update_device_info(struct ctlr_info *h,
>>>>  		sizeof(this_device->device_id));
>>>>  
>>>>  	if (this_device->devtype == TYPE_DISK &&
>>>> -		is_logical_dev_addr_mode(scsi3addr))
>>>> +		is_logical_dev_addr_mode(scsi3addr)) {
>>>>  		hpsa_get_raid_level(h, scsi3addr, &this_device->raid_level);
>>>> -	else
>>>> +		this_device->format_in_progress =
>>>> +			hpsa_format_in_progress(h, scsi3addr);
>>>> +	} else {
>>>>  		this_device->raid_level = RAID_UNKNOWN;
>>>> +		this_device->format_in_progress = 0;
>>>> +	}
>>>>  
>>>>  	if (is_OBDR_device) {
>>>>  		/* See if this is a One-Button-Disaster-Recovery device
>>>> diff --git a/drivers/scsi/hpsa.h b/drivers/scsi/hpsa.h
>>>> index bc85e72..4fd0d45 100644
>>>> --- a/drivers/scsi/hpsa.h
>>>> +++ b/drivers/scsi/hpsa.h
>>>> @@ -46,6 +46,7 @@ struct hpsa_scsi_dev_t {
>>>>  	unsigned char vendor[8];        /* bytes 8-15 of inquiry data */
>>>>  	unsigned char model[16];        /* bytes 16-31 of inquiry data */
>>>>  	unsigned char raid_level;	/* from inquiry page 0xC1 */
>>>> +	unsigned char format_in_progress;
>>>>  };
>>>>  
>>>>  struct reply_pool {
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html