qla2xxx: issues when creating a new target port

qla2xxx: issues when creating a new target port

[Chris Boot]

Hi all,

When creating a new target port in targetcli (/qla2xxx create 21:0x:...), and I have nothing plugged-in to the port, the creation process takes quite some time. After a few seconds, the kernel prints a message saying "Cable is unplugged...". So far so good.

It seems that when the target is created, it comes up in the 'enabled' state, regardless of me having set 'auto_enable_tpgt=false'. Running 'disable' from targetcli seems to try to bring the link up again ("Performing ISP error recovery" then "Cable is unplugged...") and the target remains in the enabled state.

If I then add LUNs and ACLs to the target, everything appears OK, but plugging the target port into an initiator gives odd behaviour. The initiator spends a very long time scanning the target and eventually times out and finds no LUNs. The only way to get the port working again is to save the configuration and reboot the target server, which is very frustrating.

I'm using targetcli as found in Debian wheezy with kernel 3.8.0, though I have had this problem for a little while now...

Cheers,
Chris

-- 
Chris Boot
bootc@bootc.net

Re: qla2xxx: issues when creating a new target port

[Nicholas A. Bellinger]

Hi Chris,

On Sun, 2013-03-03 at 13:49 +0000, Chris Boot wrote:
> Hi all,
> 
> When creating a new target port in targetcli (/qla2xxx create
> 21:0x:...), and I have nothing plugged-in to the port, the creation
> process takes quite some time. After a few seconds, the kernel prints
> a message saying "Cable is unplugged...". So far so good.
> 

AFAIK the delay is expected here when no physical link is detected.

> It seems that when the target is created, it comes up in the 'enabled'
> state, regardless of me having set 'auto_enable_tpgt=false'. Running
> 'disable' from targetcli seems to try to bring the link up again
> ("Performing ISP error recovery" then "Cable is unplugged...") and the
> target remains in the enabled state.
> 

Running 'disable' from targetcli will perform:

  echo 0 > /sys/kernel/config/target/qla2xxx/$FC_WWPN/$TPGT/enable

that will effectively disable target mode on the $FC_WWPN port in
question, regardless of what else (LUNs, ACLs) are configured.

Target mode is only re-enabled at the HW level for the port with:

  echo 1 > /sys/kernel/config/target/qla2xxx/$FC_WWPN/$TPGT/enable


> If I then add LUNs and ACLs to the target, everything appears OK, but
> plugging the target port into an initiator gives odd behaviour. The
> initiator spends a very long time scanning the target and eventually
> times out and finds no LUNs. The only way to get the port working
> again is to save the configuration and reboot the target server, which
> is very frustrating.
> 

Can you double check the value of ../qla2xxx/$FC_WWPN/$TPGT/enable when
this happens..?  From the description above it sounds like your
explicitly disabling target mode on the port.

> I'm using targetcli as found in Debian wheezy with kernel 3.8.0,
> though I have had this problem for a little while now...
> 

For reference, can you provide the HBAs that your reproducing with..?

Also, a target side log with

   modprobe qla2xxx ql2xextended_error_logging=0x7fffffff

would be helpful as well.

Thanks,

--nab

Re: qla2xxx: issues when creating a new target port

[Chris Boot]

On 05/03/2013 21:52, Nicholas A. Bellinger wrote:
> Hi Chris,
>
> On Sun, 2013-03-03 at 13:49 +0000, Chris Boot wrote:
>> Hi all,
>>
>> When creating a new target port in targetcli (/qla2xxx create
>> 21:0x:...), and I have nothing plugged-in to the port, the creation
>> process takes quite some time. After a few seconds, the kernel prints
>> a message saying "Cable is unplugged...". So far so good.
>>
> AFAIK the delay is expected here when no physical link is detected.

Yes, I expected the delay at this point.

>> It seems that when the target is created, it comes up in the 'enabled'
>> state, regardless of me having set 'auto_enable_tpgt=false'. Running
>> 'disable' from targetcli seems to try to bring the link up again
>> ("Performing ISP error recovery" then "Cable is unplugged...") and the
>> target remains in the enabled state.
>>
> Running 'disable' from targetcli will perform:
>
>   echo 0 > /sys/kernel/config/target/qla2xxx/$FC_WWPN/$TPGT/enable
>
> that will effectively disable target mode on the $FC_WWPN port in
> question, regardless of what else (LUNs, ACLs) are configured.
>
> Target mode is only re-enabled at the HW level for the port with:
>
>   echo 1 > /sys/kernel/config/target/qla2xxx/$FC_WWPN/$TPGT/enable

I can't easily perform this test again now, maybe I can do it later this
weekend. From memory, though, running 'disable' in targetcli _or_
running 'echo 0 >
/sys/kernel/config/target/qla2xxx/$FC_WWPN/$TPGT/enable' both indeed
have the same effect.

>> If I then add LUNs and ACLs to the target, everything appears OK, but
>> plugging the target port into an initiator gives odd behaviour. The
>> initiator spends a very long time scanning the target and eventually
>> times out and finds no LUNs. The only way to get the port working
>> again is to save the configuration and reboot the target server, which
>> is very frustrating.
>>
> Can you double check the value of ../qla2xxx/$FC_WWPN/$TPGT/enable when
> this happens..?  From the description above it sounds like your
> explicitly disabling target mode on the port.

Again, from memory, after doing a 'disable' or 'echo 0 > .../enable'
results in the 'enable' file still returning value 1. I also seem to
remember, when trying to unload the qla2xxx module, getting odd
behaviour and OOPSes - though I would want to double check this. I had
the same behaviour with two qla2xxx HBAs (one 2460 and one 2464).

>> I'm using targetcli as found in Debian wheezy with kernel 3.8.0,
>> though I have had this problem for a little while now...
>>
> For reference, can you provide the HBAs that your reproducing with..?
>
> Also, a target side log with
>
>    modprobe qla2xxx ql2xextended_error_logging=0x7fffffff
>
> would be helpful as well.

I'll see if I can get enough hardware and time together to test this
over the weekend to confirm everything I said above and gather the logs
with the error logging enabled. I'll keep you posted.

Cheers,
Chris

-- 
Chris Boot
bootc@bootc.net

Re: qla2xxx: issues when creating a new target port

[Chris Boot]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

On 05/03/13 22:15, Chris Boot wrote:
> I'll see if I can get enough hardware and time together to test this
> over the weekend to confirm everything I said above and gather the logs
> with the error logging enabled. I'll keep you posted.

I have run some tests against 3.8.2, and attached the logs.

I can only replicate the problems I ran into with the following sequence 
of events:

1. Link on the qla246x target must be down to start with (tested with 
QLA2460 and QLA2464).
2. /qla2xxx create 21:00:00:1b:32:11:a8:24
3. disable
4. luns/ create /backstores/iblock/test
5. acls/ create 10:00:00:06:2b:11:3f:a1
6. enable
7. Plug in the link to an initiator.

Even with 'set global auto_enable_tpgt=false', at step 3 the port comes 
up enabled. It seems I remembered wrongly that disabling the target port 
does not successfully disable it:

jones-hdd bootc # cat 
/sys/kernel/config/target/qla2xxx/21\:00\:00\:1b\:32\:11\:a8\:24/tpgt_1/enable 

1
jones-hdd bootc # echo 0 > 
/sys/kernel/config/target/qla2xxx/21\:00\:00\:1b\:32\:11\:a8\:24/tpgt_1/enable
jones-hdd bootc # cat 
/sys/kernel/config/target/qla2xxx/21\:00\:00\:1b\:32\:11\:a8\:24/tpgt_1/enable 

0

Once the target is configured with a LUN and ACL, enabled, and the link 
brought up by an initiator, the initiator fails to see any LUNs and 
eventually times out. The debug logging shows interesting lines such as:

Mar  8 21:06:59 jones-hdd kernel: [  183.441093] qla2xxx 
[0000:01:00.0]-f821:4: New command while device ffff8801a1b51400 is 
shutting down
Mar  8 21:06:59 jones-hdd kernel: [  183.441096] qla2xxx 
[0000:01:00.0]-e859:4: qla_target: Unable to send command to target for 
req, ignoring.

I definitely think there is a bug in there, and it feels like it's 
triggered by disabling the target port with the link down - but I'm only 
guessing at this point.

Cheers,
Chris

-- 
Chris Boot
bootc@bootc.net


[-- Attachment #2: kern.log.gz --]
[-- Type: application/x-gzip, Size: 67275 bytes --]

Re: qla2xxx: issues when creating a new target port

[Chris Boot]

On 08/03/2013 21:25, Chris Boot wrote:
> On 05/03/13 22:15, Chris Boot wrote:
>> I'll see if I can get enough hardware and time together to test this
>> over the weekend to confirm everything I said above and gather the logs
>> with the error logging enabled. I'll keep you posted.
>
> I have run some tests against 3.8.2, and attached the logs.
>
> I can only replicate the problems I ran into with the following 
> sequence of events:
>
> 1. Link on the qla246x target must be down to start with (tested with 
> QLA2460 and QLA2464).
> 2. /qla2xxx create 21:00:00:1b:32:11:a8:24
> 3. disable
> 4. luns/ create /backstores/iblock/test
> 5. acls/ create 10:00:00:06:2b:11:3f:a1
> 6. enable
> 7. Plug in the link to an initiator.
>
> Even with 'set global auto_enable_tpgt=false', at step 3 the port 
> comes up enabled. It seems I remembered wrongly that disabling the 
> target port does not successfully disable it:
>
> jones-hdd bootc # cat 
> /sys/kernel/config/target/qla2xxx/21\:00\:00\:1b\:32\:11\:a8\:24/tpgt_1/enable 
>
> 1
> jones-hdd bootc # echo 0 > 
> /sys/kernel/config/target/qla2xxx/21\:00\:00\:1b\:32\:11\:a8\:24/tpgt_1/enable
> jones-hdd bootc # cat 
> /sys/kernel/config/target/qla2xxx/21\:00\:00\:1b\:32\:11\:a8\:24/tpgt_1/enable 
>
> 0
>
> Once the target is configured with a LUN and ACL, enabled, and the 
> link brought up by an initiator, the initiator fails to see any LUNs 
> and eventually times out. The debug logging shows interesting lines 
> such as:
>
> Mar  8 21:06:59 jones-hdd kernel: [  183.441093] qla2xxx 
> [0000:01:00.0]-f821:4: New command while device ffff8801a1b51400 is 
> shutting down
> Mar  8 21:06:59 jones-hdd kernel: [  183.441096] qla2xxx 
> [0000:01:00.0]-e859:4: qla_target: Unable to send command to target 
> for req, ignoring.
>
> I definitely think there is a bug in there, and it feels like it's 
> triggered by disabling the target port with the link down - but I'm 
> only guessing at this point.

Hi folks,

Has anyone had a chance to look at the above?

Cheers,
Chris

-- 
Chris Boot
bootc@bootc.net