[LSF/MM/BPF TOPIC] Multipath bio vs. request

[LSF/MM/BPF TOPIC] Multipath bio vs. request

[John Meneghini]

I will be presenting on this topic at LSF/MM/BPF this year, in the IO track.

Here's an introduction for my talk.

DMMP currently supports two different kernel IO interfaces: the BIO interface[1] (struct bio) and the Request interface[2] (struct request).
By default DMMP uses the Request interface and over the years much work has been done test and improve the performance of the DMMP Request
interface. DMMP can also be manually configured to use the BIO interface. The DMMP BIO interface is supported but little work has been done
to test and improve its performance. DMMP is currently the only upstream component which continues to use the Request interface for submitting IO.

At the ALPSS 2024 conference last October we discussed the possibility of deprecating and eventually removing support the Request interface
as kernel API. Such a change could impact DMMP so I was asked if Red Hat would be willing to support the effort by measuring the performance
of DMMP's BIO interface[3] and comparing it to its Request based performance. Having such a comparative performance analysis would be very helpful
in determining what further changes might be needed to move DMMP away from using the Request interface. This would help with the overall effort
to improve BIO interface performance and eventually remove support for Request based IO as a kernel API.

In this presentation I will share the preliminary results of Red Hat's DMMP BIO vs Request performance tests[4] and discuss what the next possible
steps could be for moving forward.

The tests and performance graphs in this presentation were developed and run by Samuel Petrovic <spetrovi@redhat.com>.
Credit goes to Samuel for creating these performance tests and many thanks to Benjamin Marzinski <bmarzins@redhat.com>,
Mikulas Patocka <mpatocka@redhat.com> and others on the Red Hat DMMP and Performance teams who contributed to this work.

[1] https://lwn.net/Articles/736534/
[2] https://lwn.net/Articles/738449/
[3] https://lore.kernel.org/linux-scsi/643e61a8-b0cb-4c9d-831a-879aa86d888e@redhat.com
[4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/

-- 
John A. Meneghini
Senior Principal Platform Storage Engineer
RHEL SST - Platform Storage Group
jmeneghi@redhat.com

Re: [LSF/MM/BPF TOPIC] Multipath bio vs. request

[Mike Snitzer]

Hi John,

On Sat, Mar 22, 2025 at 02:38:29PM -0400, John Meneghini wrote:
> I will be presenting on this topic at LSF/MM/BPF this year, in the IO track.
> 
> Here's an introduction for my talk.
> 
> DMMP currently supports two different kernel IO interfaces: the BIO interface[1] (struct bio) and the Request interface[2] (struct request).
> By default DMMP uses the Request interface and over the years much work has been done test and improve the performance of the DMMP Request
> interface. DMMP can also be manually configured to use the BIO interface. The DMMP BIO interface is supported but little work has been done
> to test and improve its performance. DMMP is currently the only upstream component which continues to use the Request interface for submitting IO.

As I clarified at lunch today, your "DMMP is currently the only
upstream component which continues to use the Request interface for
submitting IO." makes no sense to me.  The request-based DM multipath
target is a blk-mq driver.  It just acts like most blk-mq drivers.

What is different is DM core's request-based code will clone each
request that gets submitted to the request-based DMMP device.  And
then when the request is submitted to an underlying path it gets
directly inserted in the unlering blk-mq request-queue for that path.

So in those aspects request-based DM core and DM multipath are unique
and they do require block interfaces that only benefit DMMP -- but
that has _always_ been the case (nothing else ever needed to clone
requests before submitting them).

> At the ALPSS 2024 conference last October we discussed the possibility of deprecating and eventually removing support the Request interface
> as kernel API. Such a change could impact DMMP so I was asked if Red Hat would be willing to support the effort by measuring the performance
> of DMMP's BIO interface[3] and comparing it to its Request based performance. Having such a comparative performance analysis would be very helpful
> in determining what further changes might be needed to move DMMP away from using the Request interface. This would help with the overall effort
> to improve BIO interface performance and eventually remove support for Request based IO as a kernel API.
> 
> In this presentation I will share the preliminary results of Red Hat's DMMP BIO vs Request performance tests[4] and discuss what the next possible
> steps could be for moving forward.
> 
> The tests and performance graphs in this presentation were developed and run by Samuel Petrovic <spetrovi@redhat.com>.
> Credit goes to Samuel for creating these performance tests and many thanks to Benjamin Marzinski <bmarzins@redhat.com>,
> Mikulas Patocka <mpatocka@redhat.com> and others on the Red Hat DMMP and Performance teams who contributed to this work.
> 
> [1] https://lwn.net/Articles/736534/
> [2] https://lwn.net/Articles/738449/
> [3] https://lore.kernel.org/linux-scsi/643e61a8-b0cb-4c9d-831a-879aa86d888e@redhat.com
> [4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/

Other useful context is the 2007 paper that provides an overview of
why dm-multipath was switched from bio-based to request-based:
https://www.kernel.org/doc/ols/2007/ols2007v2-pages-235-244.pdf

Re: [LSF/MM/BPF TOPIC] Multipath bio vs. request

[Mike Snitzer]

On Mon, Mar 24, 2025 at 04:20:12PM -0400, Mike Snitzer wrote:
> Hi John,
> 
> On Sat, Mar 22, 2025 at 02:38:29PM -0400, John Meneghini wrote:
> > I will be presenting on this topic at LSF/MM/BPF this year, in the IO track.
> > 
> > Here's an introduction for my talk.
> > 
> > DMMP currently supports two different kernel IO interfaces: the BIO interface[1] (struct bio) and the Request interface[2] (struct request).
> > By default DMMP uses the Request interface and over the years much work has been done test and improve the performance of the DMMP Request
> > interface. DMMP can also be manually configured to use the BIO interface. The DMMP BIO interface is supported but little work has been done
> > to test and improve its performance. DMMP is currently the only upstream component which continues to use the Request interface for submitting IO.
> 
> As I clarified at lunch today, your "DMMP is currently the only
> upstream component which continues to use the Request interface for
> submitting IO." makes no sense to me.  The request-based DM multipath
> target is a blk-mq driver.  It just acts like most blk-mq drivers.
> 
> What is different is DM core's request-based code will clone each
> request that gets submitted to the request-based DMMP device.  And
> then when the request is submitted to an underlying path it gets
> directly inserted in the unlering blk-mq request-queue for that path.

Sorry for typoe: s/unlering/underlying/
 
> So in those aspects request-based DM core and DM multipath are unique
> and they do require block interfaces that only benefit DMMP -- but
> that has _always_ been the case (nothing else ever needed to clone
> requests before submitting them).
> 
> > At the ALPSS 2024 conference last October we discussed the possibility of deprecating and eventually removing support the Request interface
> > as kernel API. Such a change could impact DMMP so I was asked if Red Hat would be willing to support the effort by measuring the performance
> > of DMMP's BIO interface[3] and comparing it to its Request based performance. Having such a comparative performance analysis would be very helpful
> > in determining what further changes might be needed to move DMMP away from using the Request interface. This would help with the overall effort
> > to improve BIO interface performance and eventually remove support for Request based IO as a kernel API.
> > 
> > In this presentation I will share the preliminary results of Red Hat's DMMP BIO vs Request performance tests[4] and discuss what the next possible
> > steps could be for moving forward.
> > 
> > The tests and performance graphs in this presentation were developed and run by Samuel Petrovic <spetrovi@redhat.com>.
> > Credit goes to Samuel for creating these performance tests and many thanks to Benjamin Marzinski <bmarzins@redhat.com>,
> > Mikulas Patocka <mpatocka@redhat.com> and others on the Red Hat DMMP and Performance teams who contributed to this work.
> > 
> > [1] https://lwn.net/Articles/736534/
> > [2] https://lwn.net/Articles/738449/
> > [3] https://lore.kernel.org/linux-scsi/643e61a8-b0cb-4c9d-831a-879aa86d888e@redhat.com
> > [4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/
> 
> Other useful context is the 2007 paper that provides an overview of
> why dm-multipath was switched from bio-based to request-based:
> https://www.kernel.org/doc/ols/2007/ols2007v2-pages-235-244.pdf
> 

Re: [LSF/MM/BPF TOPIC] Multipath bio vs. request

[Mike Christie]

On 3/22/25 1:38 PM, John Meneghini wrote:
> [4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/

For the tests run in the link, what did you use for dm_mq_nr_hw_queues
and dm_mq_queue_depth?