[PATCH] migration/rdma: Allow building without on-demand paging support

[PATCH] migration/rdma: Allow building without on-demand paging support

[Philippe Mathieu-Daudé]

On-demand paging support was added in libibverbs v1.2.0 in
commit https://github.com/linux-rdma/rdma-core/commit/e500adc7b1

We don't check the libibverbs, so add a meson check on the
IBV_ACCESS_ON_DEMAND symbol, and define HAVE_IBV_ACCESS_ON_DEMAND
if found. Restrict rdma_support_odp() so it returns %false when
on-demand paging is not supported.

This fixes:

  migration/rdma.c: In function 'rdma_support_odp':
  migration/rdma.c:1133:12: error: variable 'attr' has initializer but incomplete type
   1133 |     struct ibv_device_attr_ex attr = {0};
        |            ^~~~~~~~~~~~~~~~~~
  migration/rdma.c:1135:9: warning: implicit declaration of function 'ibv_query_device_ex'; did you mean 'ibv_query_device'? [-Wimplicit-function-declaration]
   1135 |     if (ibv_query_device_ex(dev, NULL, &attr)) {
        |         ^~~~~~~~~~~~~~~~~~~
        |         ibv_query_device
  migration/rdma.c:1135:9: warning: nested extern declaration of 'ibv_query_device_ex' [-Wnested-externs]
  migration/rdma.c:1139:38: error: 'IBV_ODP_SUPPORT' undeclared (first use in this function); did you mean 'IBV_QP_PORT'?
   1139 |     if (attr.odp_caps.general_caps & IBV_ODP_SUPPORT) {
        |                                      ^~~~~~~~~~~~~~~
        |                                      IBV_QP_PORT
  migration/rdma.c: In function 'qemu_rdma_reg_whole_ram_blocks':
  migration/rdma.c:1189:27: error: 'IBV_ACCESS_ON_DEMAND' undeclared (first use in this function); did you mean 'IBV_ACCESS_MW_BIND'?
   1189 |                 access |= IBV_ACCESS_ON_DEMAND;
        |                           ^~~~~~~~~~~~~~~~~~~~
        |                           IBV_ACCESS_MW_BIND
  ninja: build stopped: subcommand failed.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 meson.build      | 3 +++
 migration/rdma.c | 6 ++++++
 2 files changed, 9 insertions(+)

diff --git a/meson.build b/meson.build
index 837a2bdb56..7c6436ac9e 100644
--- a/meson.build
+++ b/meson.build
@@ -2410,6 +2410,9 @@ if rdma.found()
                        cc.has_function('ibv_advise_mr',
                                        dependencies: rdma,
                                        prefix: '#include <infiniband/verbs.h>'))
+  config_host_data.set('HAVE_IBV_ACCESS_ON_DEMAND',
+                       cc.has_header_symbol('infiniband/verbs.h',
+                                            'IBV_ACCESS_ON_DEMAND'))
 endif
 
 have_asan_fiber = false
diff --git a/migration/rdma.c b/migration/rdma.c
index 855753c671..4717fb3143 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1127,9 +1127,14 @@ static int qemu_rdma_alloc_qp(RDMAContext *rdma)
     return 0;
 }
 
+#ifndef HAVE_IBV_ACCESS_ON_DEMAND
+#define IBV_ACCESS_ON_DEMAND 0
+#endif
+
 /* Check whether On-Demand Paging is supported by RDAM device */
 static bool rdma_support_odp(struct ibv_context *dev)
 {
+#ifdef HAVE_IBV_ACCESS_ON_DEMAND
     struct ibv_device_attr_ex attr = {0};
 
     if (ibv_query_device_ex(dev, NULL, &attr)) {
@@ -1139,6 +1144,7 @@ static bool rdma_support_odp(struct ibv_context *dev)
     if (attr.odp_caps.general_caps & IBV_ODP_SUPPORT) {
         return true;
     }
+#endif
 
     return false;
 }
-- 
2.41.0

Re: [PATCH] migration/rdma: Allow building without on-demand paging support

[Daniel P. Berrangé]

On Thu, May 02, 2024 at 11:05:47AM +0200, Philippe Mathieu-Daudé wrote:
> On-demand paging support was added in libibverbs v1.2.0 in
> commit https://github.com/linux-rdma/rdma-core/commit/e500adc7b1

That is 9 years old, so I'm surprised any distro we target still
is so outdated. Can you say what distro you're seeing a problem
on and what version it has ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] migration/rdma: Allow building without on-demand paging support

[Philippe Mathieu-Daudé]

On 2/5/24 11:10, Daniel P. Berrangé wrote:
> On Thu, May 02, 2024 at 11:05:47AM +0200, Philippe Mathieu-Daudé wrote:
>> On-demand paging support was added in libibverbs v1.2.0 in
>> commit https://github.com/linux-rdma/rdma-core/commit/e500adc7b1
> 
> That is 9 years old, so I'm surprised any distro we target still
> is so outdated. Can you say what distro you're seeing a problem
> on and what version it has ?

This is Oracle Solaris 11.4 SRU, released 2 weeks ago:
https://support.oracle.com/knowledge/Sun%20Microsystems/2433412_1.html

I'm not sure how to detect the version, I'm seeing downstream
patches applied on top.

Re: [PATCH] migration/rdma: Allow building without on-demand paging support

[Philippe Mathieu-Daudé]

On 2/5/24 11:19, Philippe Mathieu-Daudé wrote:
> On 2/5/24 11:10, Daniel P. Berrangé wrote:
>> On Thu, May 02, 2024 at 11:05:47AM +0200, Philippe Mathieu-Daudé wrote:
>>> On-demand paging support was added in libibverbs v1.2.0 in
>>> commit https://github.com/linux-rdma/rdma-core/commit/e500adc7b1
>>
>> That is 9 years old, so I'm surprised any distro we target still
>> is so outdated. Can you say what distro you're seeing a problem
>> on and what version it has ?
> 
> This is Oracle Solaris 11.4 SRU, released 2 weeks ago:
> https://support.oracle.com/knowledge/Sun%20Microsystems/2433412_1.html
> 
> I'm not sure how to detect the version, I'm seeing downstream
> patches applied on top.

Regardless, we can't expect any rdma library version to work,
either we ask for some version, or we check that symbol is there
and reject if not as "your rdma is too old", so the user can
choose to build with --disable-rdma.

Re: [PATCH] migration/rdma: Allow building without on-demand paging support

[Daniel P. Berrangé]

On Thu, May 02, 2024 at 11:21:31AM +0200, Philippe Mathieu-Daudé wrote:
> On 2/5/24 11:19, Philippe Mathieu-Daudé wrote:
> > On 2/5/24 11:10, Daniel P. Berrangé wrote:
> > > On Thu, May 02, 2024 at 11:05:47AM +0200, Philippe Mathieu-Daudé wrote:
> > > > On-demand paging support was added in libibverbs v1.2.0 in
> > > > commit https://github.com/linux-rdma/rdma-core/commit/e500adc7b1
> > > 
> > > That is 9 years old, so I'm surprised any distro we target still
> > > is so outdated. Can you say what distro you're seeing a problem
> > > on and what version it has ?
> > 
> > This is Oracle Solaris 11.4 SRU, released 2 weeks ago:
> > https://support.oracle.com/knowledge/Sun%20Microsystems/2433412_1.html
> > 
> > I'm not sure how to detect the version, I'm seeing downstream
> > patches applied on top.
> 
> Regardless, we can't expect any rdma library version to work,
> either we ask for some version, or we check that symbol is there
> and reject if not as "your rdma is too old", so the user can
> choose to build with --disable-rdma.

The RDMA libs appear to have pkg-config files, so I agree that we
should be requesting a min version in meson that matches what our
code actually requires.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] migration/rdma: Allow building without on-demand paging support

[Daniel P. Berrangé]

On Thu, May 02, 2024 at 11:19:28AM +0200, Philippe Mathieu-Daudé wrote:
> On 2/5/24 11:10, Daniel P. Berrangé wrote:
> > On Thu, May 02, 2024 at 11:05:47AM +0200, Philippe Mathieu-Daudé wrote:
> > > On-demand paging support was added in libibverbs v1.2.0 in
> > > commit https://github.com/linux-rdma/rdma-core/commit/e500adc7b1
> > 
> > That is 9 years old, so I'm surprised any distro we target still
> > is so outdated. Can you say what distro you're seeing a problem
> > on and what version it has ?
> 
> This is Oracle Solaris 11.4 SRU, released 2 weeks ago:
> https://support.oracle.com/knowledge/Sun%20Microsystems/2433412_1.html

Solaris 11.4 was originally from 2018 IIUC, so that explains why
it is so ancient.

> I'm not sure how to detect the version, I'm seeing downstream
> patches applied on top.

When this was first introduced there was no pkg-config files
present, nor was it using cmake, or even library versioning
afaics.

The pkgconfig files were added 3 years later in

  commit df5fe3c2fa9d7dbb38fb7b4878955330620211ed
  Author: Luca Boccassi <bluca@debian.org>
  Date:   Tue Aug 28 19:46:07 2018 +0100

    Generate and install pkgconfig files for libs

which at that point had defined:

  CMakeLists.txt:set(PACKAGE_VERSION "20.0")
  librdmacm/CMakeLists.txt:  1 1.1.${PACKAGE_VERSION}
  libibverbs/CMakeLists.txt:  1 1.5.${PACKAGE_VERSION}

So I think we can try checking

  libibverbs >= 1.5
  rdmacm >= 1.1

in meson.build, and see if that solves the Solaris
problem....depends what the scope of their downstream
patches is.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|