Status of SCSI referrals support in dm-multipath

Status of SCSI referrals support in dm-multipath

[Sebastian Herbszt]

Hello,

I am trying to find out the status of SCSI referrals support in dm-multipath.
SCSI referrals support was a topic at LSF [1] but I wasn't able to find any
information. Back in 2010 Hannes suggested the use of multiple entries in a
multipath table [2] but this doesn't seem to work (anymore):

[ 7750.354190] device-mapper: table: Request-based dm doesn't support multiple targets yet

Is there any SCSI referrals support yet?

[1] http://marc.info/?l=linux-scsi&m=129553880112460&w=2
[2] http://www.redhat.com/archives/dm-devel/2010-June/msg00155.html

Sebastian

Re: Status of SCSI referrals support in dm-multipath

[Hannes Reinecke]

On 06/01/2014 10:46 PM, Sebastian Herbszt wrote:
> Hello,
>
> I am trying to find out the status of SCSI referrals support in dm-multipath.
> SCSI referrals support was a topic at LSF [1] but I wasn't able to find any
> information. Back in 2010 Hannes suggested the use of multiple entries in a
> multipath table [2] but this doesn't seem to work (anymore):
>
> [ 7750.354190] device-mapper: table: Request-based dm doesn't support multiple targets yet
>
> Is there any SCSI referrals support yet?
>
> [1] http://marc.info/?l=linux-scsi&m=129553880112460&w=2
> [2] http://www.redhat.com/archives/dm-devel/2010-June/msg00155.html
>
Not as of now. I've managed to push referrals support into 
target_core, so you now can set up a backend which supports referrals.

And for the multiple entries I've though to use a linear target on 
top of the multipath target.
The linear target would just split up the device in chunks, 
according to the referrals layout.
And then you would have several multipath devices (all referring to 
the same disk), one for each supported referrals ALUA configuration.
The linear target would then map each chunk to the correct multipath 
device.

It'll be a beast for failover, but should be possible.
And maybe it's better to use dm-switch here; the number or referrals 
might overflow dm-linear ...

Cheers,

Hannes

-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: Status of SCSI referrals support in dm-multipath

[Sebastian Herbszt]

Hannes Reinecke wrote:
> On 06/01/2014 10:46 PM, Sebastian Herbszt wrote:
> > Hello,
> >
> > I am trying to find out the status of SCSI referrals support in dm-multipath.
> > SCSI referrals support was a topic at LSF [1] but I wasn't able to find any
> > information. Back in 2010 Hannes suggested the use of multiple entries in a
> > multipath table [2] but this doesn't seem to work (anymore):
> >
> > [ 7750.354190] device-mapper: table: Request-based dm doesn't support multiple targets yet
> >
> > Is there any SCSI referrals support yet?
> >
> > [1] http://marc.info/?l=linux-scsi&m=129553880112460&w=2
> > [2] http://www.redhat.com/archives/dm-devel/2010-June/msg00155.html
> >
> Not as of now. I've managed to push referrals support into 
> target_core, so you now can set up a backend which supports referrals.

I gathered the following information from a storage with referral support.

> And for the multiple entries I've though to use a linear target on 
> top of the multipath target.
> The linear target would just split up the device in chunks, 
> according to the referrals layout.

I created a new thin provisioned volume which reported

Referrals VPD page (SBC):
  User data segment size: 0
  User data segment multiplier: 0

and didn't return any user data segment referral descriptors.
Since the chunk size is not known, splitting the device is not possible.
The storage reported 4 equal target port groups (sd[b-e]). I guess the
initial setup would consist of a linear target with just a single mapping.

As soon as I started to fill the volume the storage returned the
descriptors

Report referrals:
  descriptor 0:
    target port descriptors: 4
    user data segment: first lba 0, last lba 2752511
      target port descriptor 0:
        port group 8080 state (active/non optimized)
      target port descriptor 1:
        port group 8081 state (active/non optimized)
      target port descriptor 2:
        port group 8090 state (active/optimized)
      target port descriptor 3:
        port group 8091 state (active/optimized)
  descriptor 1:
    target port descriptors: 4
    user data segment: first lba 2752512, last lba 8257535
      target port descriptor 0:
        port group 8080 state (active/optimized)
      target port descriptor 1:
        port group 8081 state (active/optimized)
      target port descriptor 2:
        port group 8090 state (active/non optimized)
      target port descriptor 3:
        port group 8091 state (active/non optimized)
  descriptor 2:
    target port descriptors: 4
    user data segment: first lba 8257536, last lba 11010047
      target port descriptor 0:
        port group 8080 state (active/non optimized)
      target port descriptor 1:
        port group 8081 state (active/non optimized)
      target port descriptor 2:
        port group 8090 state (active/optimized)
      target port descriptor 3:
        port group 8091 state (active/optimized)
...

The first descriptor covers 1344 MB; the second twice as much.
On a full volume the smallest size was 1344MB and few descriptors
covered multiple of it.
After running a rebalance operation on the storage the layout partially
changed

Report referrals:
  descriptor 0:
    target port descriptors: 4
    user data segment: first lba 0, last lba 2752511
      target port descriptor 0:
        port group 8080 state (active/optimized)
      target port descriptor 1:
        port group 8081 state (active/optimized)
      target port descriptor 2:
        port group 8090 state (active/non optimized)
      target port descriptor 3:
        port group 8091 state (active/non optimized)
  descriptor 1:
    target port descriptors: 4
    user data segment: first lba 2752512, last lba 5505023
      target port descriptor 0:
        port group 8080 state (active/non optimized)
      target port descriptor 1:
        port group 8081 state (active/non optimized)
      target port descriptor 2:
        port group 8090 state (active/optimized)
      target port descriptor 3:
        port group 8091 state (active/optimized)
  descriptor 2:
    target port descriptors: 4
    user data segment: first lba 5505024, last lba 8257535
      target port descriptor 0:
        port group 8080 state (active/optimized)
      target port descriptor 1:
        port group 8081 state (active/optimized)
      target port descriptor 2:
        port group 8090 state (active/non optimized)
      target port descriptor 3:
        port group 8091 state (active/non optimized)
  descriptor 3:
    target port descriptors: 4
    user data segment: first lba 8257536, last lba 11010047
      target port descriptor 0:
        port group 8080 state (active/non optimized)
      target port descriptor 1:
        port group 8081 state (active/non optimized)
      target port descriptor 2:
        port group 8090 state (active/optimized)
      target port descriptor 3:
        port group 8091 state (active/optimized)
...

Is there currently a way to get notified on such changes? If we start with
a single mapping we need a way to trigger a (re-)discovery/modification. 

> And then you would have several multipath devices (all referring to 
> the same disk), one for each supported referrals ALUA configuration.
> The linear target would then map each chunk to the correct multipath 
> device.

I assume the above configuration would require two multipath devices;
one for

      target port descriptor 0:
        port group 8080 state (active/optimized)
      target port descriptor 1:
        port group 8081 state (active/optimized)
      target port descriptor 2:
        port group 8090 state (active/non optimized)
      target port descriptor 3:
        port group 8091 state (active/non optimized)

and a second for

      target port descriptor 0:
        port group 8080 state (active/non optimized)
      target port descriptor 1:
        port group 8081 state (active/non optimized)
      target port descriptor 2:
        port group 8090 state (active/optimized)
      target port descriptor 3:
        port group 8091 state (active/optimized)

> It'll be a beast for failover, but should be possible.
> And maybe it's better to use dm-switch here; the number or referrals 
> might overflow dm-linear ...

Does dm-switch work with different chunk sizes? Is there a theoretical or
practical limit for dm-linear? My 100 GB device consists of 62 descriptors.
 
> Cheers,
> 
> Hannes

Sebastian