Re: [Scst-devel] Poisoning of Linux initiators on SCST reboot.

[Vladislav Bolkhovitin <vst@vlnb.net>]

Vladislav Bolkhovitin wrote:
> greg@enjellic.com wrote:
>> Good morning to everyone, hope your respective days are going well.
>> Sorry for the wide cast on this but I wanted to get what would seem to
>> be the concerned parties on this issue in the loop.
>>
>> We have been putting SCST through an extensive round of pre-production
>> testing.  I wanted to start following up on some of the issues we have
>> noted.
>>
>> We will be putting SCST into service to support mirrored storage from
>> client initiators to two separate data-centers.  The filesystems on
>> the client initiators access storage at the two data-centers via a
>> Linux MD RAID1 device.  The SAN architecture is based on Cisco
>> MDS-9509 switches.
>>
>> Just as an aside for people considering use of SCST.  The core engine
>> has been rock solid.  Our testing rounds consists of driving around
>> 1/8 of a petabyte of widely disparate I/O types from multiple
>> initiators to a pair of targets in the two data-centers.  SCST hasn't
>> missed a beat so far, so kudos to Vlad and everyone involved in its
>> development.
> 
> Thanks! Your support is very much appreciated and exactly on time, 
> because I'm going to submit SCST patches for review and inclusion into 
> the kernel next week.
> 
>> As we began forced failure testing one issue has come up that I wanted
>> to advise people of.  A hard reboot of an SCST target server results
>> in the 'poisoning' of Linux based initiators.  We verified the issue
>> as being present on client initiators running the stock RHEL5 kernel
>> up through 2.6.26.
>>
>> The targets are using Qlogic 2462 cards using the isp_mod driver.  The
>> client initiators are using Qlogic 2342 cards with the qla2xxx driver.
>>
>> Failure mode is as follows:
>>
>>         1.) Configure SCST based storage for an initiator (vdisk
>>             based).
>>
>>         2.) Activate initiator.  Initiator logs into fabric and
>>             discovers SCST based storage.
>>
>>         3.) Force SCST target failure by rebooting or pulling power.
>>
>>         4.) SCST target returns to service and logs into zone.
>>
>>         5.) Initiator picks up RSCN but re-activates the rport for
>>             SCST server as an INITIATOR rather than TARGET role.
>>
>> After this point in time the initiator is effectively 'poisoned'.
>>
>> Nothing short of unloading and reloading the Qlogic 2xxx driver on the
>> client initiator will allow the initiator to recognize the SCST server
>> as a target device.  A driver unload/reload of course is not an option
>> to restore connectivity since it would take the remaining live side of
>> the mirror off-line as well.
>>
>> We finally figured out what seems to be happening by watching the logs
>> on the client and comparing what was going on there to the FLOGI login
>> status on the fabric.
>>
>> When the SCST target server reboots the initiator times out the remote
>> port and places it into 'unknown' state.  The qla2xxx driver,
>> according to the source code, maintains the previous rport state in
>> driver internal data.
>>
>> The 2462 card in the target on boot logs into the fabric with an
>> initiator role, I'm assuming in support of BIOS based SAN booting. The
>> client initiator picks up on this and re-activates the rport as being
>> in an INITIATOR role.
> 
> You should be able to switch off this behavior by disabling the SAN 
> booting in the card's BIOS.
> 
>> Loading the isp_mod driver causes the 2462 card in the target to be
>> shutdown.  The client initiator picks up on this and times out the
>> rport retaining the last rport state as INITIATOR.
>>
>> Enabling target mode on the 2462 causes it to log back into the
>> fabric.  The client initiator picks up on the RSCN but refuses to
>> transition the rport from INITIATOR to TARGET state.  Without going
>> into TARGET state the remote port won't have SCSI device discovery
>> initiated against it and hence the SCST based storage is inaccessible.
>>
>> Activating a LIP on the client initiates a new fabric login attempt
>> which completes with the following message:
>>
>> Jul 24 02:53:59 init-test kernel: rport-2:0-0: blocked FC remote port
>> time out: no longer a FCP target, removing starget
>>
>> Which from a review of the source code seems consistent with our
>> analysis of the problem.
>>
>> The culprit is the following code from drivers/scsi/scsi_transport_fc.c:
>>
>>         if ((rport->port_state == FC_PORTSTATE_ONLINE) &&
>>             (rport->scsi_target_id != -1) &&
>>             !(rport->roles & FC_PORT_ROLE_FCP_TARGET)) {
>>                 dev_printk(KERN_ERR, &rport->dev,
>>                         "blocked FC remote port time out: no longer"
>>                         " a FCP target, removing starget\n");
>>                 spin_unlock_irqrestore(shost->host_lock, flags);
>>                 scsi_target_unblock(&rport->dev);
>>                 fc_queue_work(shost, &rport->stgt_delete_work);
>>                 return;
>>         }
>>
>> The above gets executed in response to the LIP on the initiator.  The
>> value in rport->roles is being populated with what the remote target
>> was INITIATOR rather than its current TARGET state.
>>
>> Windows client initiators running against the SCST targets get the
>> transition and login sequence correct.  When the SCST target is
>> re-activated after the cold boot those clients immediately re-discover
>> their storage while the Linux clients issue error messages about loss
>> of the remote target.
>>
>> While all this doesn't seem to be technically a bug with SCST it
>> certainly is a problematic usage scenario.  It may also explain why
>> some individuals may have had problems getting SCST clients to access
>> their storage.
>>
>> If a test SCST server was plugged into an active zone and turned on it
>> would immediately poison any Linux clients.  No amount of proper
>> configuration on the target would allow the client to access storage
>> until the client was rebooted or its drivers reloaded.
>>
>> Any suggestions on how to move forward would be appreciated.  We've
>> got a pretty extensive test environment and would be happy to test run
>> any suggested changes or patches.
> 
> I've also many times seen how Linux Qlogic qla2xxx driver "lost" remote 
> ports. But that was from the target side and I wasn't able to figure out 
> the exact test case for that. Plus, we found out a suitable for target 
> workaround: usage of INITIATOR PORT NAME field in ATIO IOCB for the lost 
> ports.
> 
> So, qla2xxx driver definitely has problem(s) in this area. The fact that 
> Windows works well in this scenario only additionally proves that. But 
> I'm afraid, you have the only way to deal with it is to fix qla2xxx 
> driver itself.

Sorry, I meant "yourself".

> My experience with contacts with Andrew Vasquez, the 
> driver's maintainer, that you need something more valuable than problems 
> with some home brewed target to make him interested. Otherwise your 
> questions will be simply ignored.
> 
>> Once again a thank you to everyone who has contributed to SCST
>> development.  Other than this and a few additional glitches I will
>> follow up with via additional e-mails it is presenting itself as a
>> very solid platform for storage delivery.
>>
>> Best wishes for a pleasant weekend to everyone.
>>
>> As always,
>> Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
>> 4206 N. 19th Ave.           Specializing in information infra-structure
>> Fargo, ND  58102            development.
>> PH: 701-281-1686
>> FAX: 701-281-3949           EMAIL: greg@enjellic.com
>> ------------------------------------------------------------------------------
>> "Much work remains to be done before we can announce our total failure
>>  to make any progress."
>>                                 -- Mike Kelly
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Scst-devel mailing list
>> Scst-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scst-devel
>>
> 
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Scst-devel mailing list
> Scst-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel
>