[LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Mike Snitzer]

Hi,

I'd like to attend LSF (and the first day of Vault).

As a DM maintainer I'm open to discussing anything DM related.  In
particular I'd like to at least have hallway track discussions with
key people interested in DM multipath support for blk-mq devices.
Keith Busch and I have gone ahead and implemented the bulk of this
request-based DM support for blk-mq devices, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-for-3.20-blk-mq

But Bart says he is seeing inconsistent performance now that the code
appears relatively stable.  I'll continue analyzing this aspect of
things and by the time LSF rolls around I hope the code to be sorted.

Another topic we need to come to terms with is the state of XCOPY
(whether the initial approach needs further work, etc) -- Mikulas
Patocka reworked aspects of Martin's initial approach but it hasn't
progressed upstream:
https://www.redhat.com/archives/dm-devel/2014-October/msg00167.html

Re: [dm-devel] [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Hannes Reinecke]

On 01/04/2015 06:16 PM, Mike Snitzer wrote:
> Hi,
> 
> I'd like to attend LSF (and the first day of Vault).
> 
> As a DM maintainer I'm open to discussing anything DM related.  In
> particular I'd like to at least have hallway track discussions with
> key people interested in DM multipath support for blk-mq devices.
> Keith Busch and I have gone ahead and implemented the bulk of this
> request-based DM support for blk-mq devices, see:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-for-3.20-blk-mq
> 
> But Bart says he is seeing inconsistent performance now that the code
> appears relatively stable.  I'll continue analyzing this aspect of
> things and by the time LSF rolls around I hope the code to be sorted.
> 
I'd be interested in having a discussion here, too.
hch and me have discussed an alternative approach for multipathing
(send the request directly and only clone request if the original
submission failed) which I'd like to discuss with a broader audience.
And I'd be interested in getting multipath to work with blk-mq, naturally.

> Another topic we need to come to terms with is the state of XCOPY
> (whether the initial approach needs further work, etc) -- Mikulas
> Patocka reworked aspects of Martin's initial approach but it hasn't
> progressed upstream:
> https://www.redhat.com/archives/dm-devel/2014-October/msg00167.html
> 
Yep. That definitely needs to be discussed.
Especially we'd need to discuss how to handle exceptions, seeing that
XCOPY might fail basically at any time.
And there are some corner-cases (bandwidth starvation on the target)
which needs to be discussed, too.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [dm-devel] [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Yep. That definitely needs to be discussed.  Especially we'd
Hannes> need to discuss how to handle exceptions, seeing that XCOPY
Hannes> might fail basically at any time.  

Like any SCSI command :)

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [dm-devel] [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Hannes Reinecke]

On 01/07/2015 12:55 AM, Martin K. Petersen wrote:
>>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:
> 
> Hannes> Yep. That definitely needs to be discussed.  Especially we'd
> Hannes> need to discuss how to handle exceptions, seeing that XCOPY
> Hannes> might fail basically at any time.  
> 
> Like any SCSI command :)
> 
Not quite. XCOPY is optional, and the system works well without it.
So the exception handling would need to copy things by hand to avoid
regressions.

Plus XCOPY requires some elaborate setup, and even if those succeeded
the array might still fail the command.
_And_ there is no guarantee that that the XCOPY command is actually
faster than the manual procedure.

I guess we'll need some timing / initialisation routine to figure
out what speed-up is to be had with XCOPY.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [dm-devel] [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Not quite. XCOPY is optional, and the system works well without
Hannes> it.  So the exception handling would need to copy things by hand
Hannes> to avoid regressions.

Or defer to user space.

But it's really no different from how we do WRITE SAME which may or may
not work. If it fails we fall back to writing zeroes.

Hannes> Plus XCOPY requires some elaborate setup, and even if those
Hannes> succeeded the array might still fail the command.  _And_ there
Hannes> is no guarantee that that the XCOPY command is actually faster
Hannes> than the manual procedure.

It saves the data a roundtrip on the fabric. That itself may be more
valuable than a direct bandwidth win if there is concurrent I/O on the
wire.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [dm-devel] [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Hannes Reinecke]

On 01/08/2015 12:39 AM, Martin K. Petersen wrote:
>>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:
> 
> Hannes> Not quite. XCOPY is optional, and the system works well without
> Hannes> it.  So the exception handling would need to copy things by hand
> Hannes> to avoid regressions.
> 
> Or defer to user space.
> 
> But it's really no different from how we do WRITE SAME which may or may
> not work. If it fails we fall back to writing zeroes.
> 
> Hannes> Plus XCOPY requires some elaborate setup, and even if those
> Hannes> succeeded the array might still fail the command.  _And_ there
> Hannes> is no guarantee that that the XCOPY command is actually faster
> Hannes> than the manual procedure.
> 
> It saves the data a roundtrip on the fabric. That itself may be more
> valuable than a direct bandwidth win if there is concurrent I/O on the
> wire.
> 
But the array might prioritize 'normal' I/O requests, and treat XCOPY
with a lower priority. So given enough load XCOPY might actually be
slower than a normal copy ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> But the array might prioritize 'normal' I/O requests, and treat
Hannes> XCOPY with a lower priority. So given enough load XCOPY might
Hannes> actually be slower than a normal copy ...

Yes, but may have lower impact on concurrent I/O.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Martin K. Petersen]

>>>>> "Mike" == Mike Snitzer <snitzer@redhat.com> writes:

Mike> Another topic we need to come to terms with is the state of XCOPY
Mike> (whether the initial approach needs further work, etc) -- Mikulas
Mike> Patocka reworked aspects of Martin's initial approach but it
Mike> hasn't progressed upstream:

Yeah, I'd like to get that sorted out too. It got stuck on running out
of request flags and that requires some block layer surgery. I'll try to
get that piece done right away for 3.20 so we can get xcopy into 3.21.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [LSF/MM ATTEND] discuss blk-mq related to DM-multipath and status of XCOPY

[Nicholas A. Bellinger]

On Sun, 2015-01-04 at 12:16 -0500, Mike Snitzer wrote:
> Hi,
> 
> I'd like to attend LSF (and the first day of Vault).
> 
> As a DM maintainer I'm open to discussing anything DM related.  In
> particular I'd like to at least have hallway track discussions with
> key people interested in DM multipath support for blk-mq devices.
> Keith Busch and I have gone ahead and implemented the bulk of this
> request-based DM support for blk-mq devices, see:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-for-3.20-blk-mq
> 
> But Bart says he is seeing inconsistent performance now that the code
> appears relatively stable.  I'll continue analyzing this aspect of
> things and by the time LSF rolls around I hope the code to be sorted.
> 
> Another topic we need to come to terms with is the state of XCOPY
> (whether the initial approach needs further work, etc) -- Mikulas
> Patocka reworked aspects of Martin's initial approach but it hasn't
> progressed upstream:
> https://www.redhat.com/archives/dm-devel/2014-October/msg00167.html

A big +1 on the host side XCOPY support bit.

--nab