libmlx5 doesn't build on s390x

libmlx5 doesn't build on s390x

[Jarod Wilson]

Not sure if it's intentional, but in my adventure of the past day or two 
to get rdma-core v13 building across all the arches I need to build it 
for, I discovered that the mlx5 bits silently don't build on s390x. No 
build failure, they just aren't attempted, so far as I can see. The only 
sign of failure was when rpm %files globs failed to locate mlx5 bits.

I noted that mlx4 has some s390x-specific trickery in 
providers/mlx4/mmio.h, and wondered if something similar was required 
for mlx5, or if "does not work on s390x and isn't expected to" is the 
norm. Currently, my builds are excluding mlx5 support on s390x due to 
the current state.

-- 
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Yishai Hadas]

On 3/23/2017 2:51 PM, Jarod Wilson wrote:
> Not sure if it's intentional, but in my adventure of the past day or two
> to get rdma-core v13 building across all the arches I need to build it
> for, I discovered that the mlx5 bits silently don't build on s390x. No
> build failure, they just aren't attempted, so far as I can see. The only
> sign of failure was when rpm %files globs failed to locate mlx5 bits.

How about mlx4, was it built successfully ? we expect the same behavior 
from build point of view as both and others providers depend on 
HAVE_COHERENT_DMA from CMakeLists.txt

Any option that you saw below message in the log ?
"Architecture NOT able to do coherent DMA (check libibverbs/arch.h) some 
providers disabled!")"

> I noted that mlx4 has some s390x-specific trickery in
> providers/mlx4/mmio.h, and wondered if something similar was required
> for mlx5, or if "does not work on s390x and isn't expected to" is the
> norm. Currently, my builds are excluding mlx5 support on s390x due to
> the current state.

The patch of mlx4 for s390x in mlx4/mmio.h came from IBM people few 
years ago, I believe that similar one is needed for mlx5 otherwise it 
won't be able to write to PCI memory, so excluding mlx5 for s390 makes 
sense. However, mlx4 should be in, is it really the case when you build 
the s390 package ?

Adding Alexey who sent the patch for mlx4 to get his input as well.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jason Gunthorpe]

On Thu, Mar 23, 2017 at 08:51:21AM -0400, Jarod Wilson wrote:
> Not sure if it's intentional, but in my adventure of the past day or two to
> get rdma-core v13 building across all the arches I need to build it for, I
> discovered that the mlx5 bits silently don't build on s390x. No build
> failure, they just aren't attempted, so far as I can see. The only sign of
> failure was when rpm %files globs failed to locate mlx5 bits.

I don't expect anything except mlx4 to *work* on s390 - fixing that
would require making a common IOMEM accessor system.

However, AFAIK, they should build..

> I noted that mlx4 has some s390x-specific trickery in providers/mlx4/mmio.h,
> and wondered if something similar was required for mlx5, or if "does not
> work on s390x and isn't expected to" is the norm. Currently, my builds are
> excluding mlx5 support on s390x due to the current state.

It would be nice if you could post logs for these various
failures..

My guess is that none of the DMA providers were compiled for S390,
which happens if the system cannot compile util/udma_barrier.h.

Perhaps this test is wrong?

#elif defined(__sparc__) || defined(__s390x__)

Maybe it should be

#elif defined(__sparc__) || defined(__s390x__) || defined(__s390__)

??

Also, you should expect mlx5 not to be built on ARM32 and in other
places where we do not support user DMA. Can you accommodate this in
the rpm spec file? Debian has the same issue too... You can simulate
this on x86-64 by patching util/udma_barrier.h to always fail to
compile.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jarod Wilson]

On 2017-03-23 12:06 PM, Yishai Hadas wrote:
> On 3/23/2017 2:51 PM, Jarod Wilson wrote:
>> Not sure if it's intentional, but in my adventure of the past day or two
>> to get rdma-core v13 building across all the arches I need to build it
>> for, I discovered that the mlx5 bits silently don't build on s390x. No
>> build failure, they just aren't attempted, so far as I can see. The only
>> sign of failure was when rpm %files globs failed to locate mlx5 bits.
>
> How about mlx4, was it built successfully ? we expect the same behavior
> from build point of view as both and others providers depend on
> HAVE_COHERENT_DMA from CMakeLists.txt
>
> Any option that you saw below message in the log ?
> "Architecture NOT able to do coherent DMA (check libibverbs/arch.h) some
> providers disabled!")"

Yep, it's there, lost it in all the verbosity. Okay, that explains that.

>> I noted that mlx4 has some s390x-specific trickery in
>> providers/mlx4/mmio.h, and wondered if something similar was required
>> for mlx5, or if "does not work on s390x and isn't expected to" is the
>> norm. Currently, my builds are excluding mlx5 support on s390x due to
>> the current state.
>
> The patch of mlx4 for s390x in mlx4/mmio.h came from IBM people few
> years ago, I believe that similar one is needed for mlx5 otherwise it
> won't be able to write to PCI memory, so excluding mlx5 for s390 makes
> sense. However, mlx4 should be in, is it really the case when you build
> the s390 package ?

Apparently not:

$ rpm -qpl libibverbs-13-1.el7.s390x.rpm
/etc/libibverbs.d
/etc/libibverbs.d/hfi1verbs.driver
/etc/libibverbs.d/ipathverbs.driver
/etc/libibverbs.d/rxe.driver
/usr/bin/rxe_cfg
/usr/lib64/libibverbs
/usr/lib64/libibverbs.so.1
/usr/lib64/libibverbs.so.1.1.13
/usr/lib64/libibverbs/libhfi1verbs-rdmav2.so
/usr/lib64/libibverbs/libipathverbs-rdmav2.so
/usr/lib64/libibverbs/librxe-rdmav2.so
/usr/share/doc/rdma-core-13/libibverbs.md
/usr/share/doc/rdma-core-13/rxe.md
/usr/share/man/man7/rxe.7.gz
/usr/share/man/man8/rxe_cfg.8.gz

Looks like only rxe, ipathverbs and hfi1verbs got built. Didn't realize 
that many others (including mlx4) weren't getting built until now. Oops.

-- 
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jarod Wilson]

On 2017-03-23 12:10 PM, Jason Gunthorpe wrote:
> On Thu, Mar 23, 2017 at 08:51:21AM -0400, Jarod Wilson wrote:
>> Not sure if it's intentional, but in my adventure of the past day or two to
>> get rdma-core v13 building across all the arches I need to build it for, I
>> discovered that the mlx5 bits silently don't build on s390x. No build
>> failure, they just aren't attempted, so far as I can see. The only sign of
>> failure was when rpm %files globs failed to locate mlx5 bits.
>
> I don't expect anything except mlx4 to *work* on s390 - fixing that
> would require making a common IOMEM accessor system.
>
> However, AFAIK, they should build..

See reply to Yishai. Not much is actually building on s390x. (Note: I'm 
not building for s390 sans-x at all).

>> I noted that mlx4 has some s390x-specific trickery in providers/mlx4/mmio.h,
>> and wondered if something similar was required for mlx5, or if "does not
>> work on s390x and isn't expected to" is the norm. Currently, my builds are
>> excluding mlx5 support on s390x due to the current state.
>
> It would be nice if you could post logs for these various
> failures..

Yeah, I can do that.

> My guess is that none of the DMA providers were compiled for S390,
> which happens if the system cannot compile util/udma_barrier.h.
>
> Perhaps this test is wrong?
>
> #elif defined(__sparc__) || defined(__s390x__)
>
> Maybe it should be
>
> #elif defined(__sparc__) || defined(__s390x__) || defined(__s390__)
>
> ??

Not sure. From rpm's perspective, I'm trying arch s390x, but maybe gcc 
is actually calling it s390.

> Also, you should expect mlx5 not to be built on ARM32 and in other
> places where we do not support user DMA. Can you accommodate this in
> the rpm spec file? Debian has the same issue too... You can simulate
> this on x86-64 by patching util/udma_barrier.h to always fail to
> compile.

We don't target arm32 at all, only arm64 (aarch64 in rpm parlance). I 
believe an 'ExcludeArch: %{arm} s390' is currently in my local spec. (I 
have a few tweaks to push up here still).

-- 
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jason Gunthorpe]

On Thu, Mar 23, 2017 at 12:50:14PM -0400, Jarod Wilson wrote:

> >Any option that you saw below message in the log ?
> >"Architecture NOT able to do coherent DMA (check libibverbs/arch.h) some
> >providers disabled!")"
> 
> Yep, it's there, lost it in all the verbosity. Okay, that explains that.

FWIW, that section after 'Missing Optional Items:' reflects everything
'wrong' with the build environment. An ideal build will list nothing,
eg on FC25.

Everything else means something is broken and cmake is working around
the breakage somehow. It is worth reviewing if the thing is really
broken or not...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jason Gunthorpe]

On Thu, Mar 23, 2017 at 12:57:37PM -0400, Jarod Wilson wrote:
> Not sure. From rpm's perspective, I'm trying arch s390x, but maybe gcc is
> actually calling it s390.

The output from build/CMakeFiles/CMakeError.log might help

As well as this output:

$ gcc -dM -E - < /dev/null

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jarod Wilson]

On 2017-03-23 1:02 PM, Jason Gunthorpe wrote:
> On Thu, Mar 23, 2017 at 12:57:37PM -0400, Jarod Wilson wrote:
>> Not sure. From rpm's perspective, I'm trying arch s390x, but maybe gcc is
>> actually calling it s390.
>
> The output from build/CMakeFiles/CMakeError.log might help
>
> As well as this output:
>
> $ gcc -dM -E - < /dev/null

Unfortunately, this is sort of a black box situation for me at the 
moment. I'm submitting the builds to our build system, which hands the 
build job off to all relevant architectures. I can probably get a 
reservation on an s390x system and poke manually though.

-- 
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: libmlx5 doesn't build on s390x

[Jarod Wilson]

On 2017-03-23 12:58 PM, Jason Gunthorpe wrote:
> On Thu, Mar 23, 2017 at 12:50:14PM -0400, Jarod Wilson wrote:
>
>>> Any option that you saw below message in the log ?
>>> "Architecture NOT able to do coherent DMA (check libibverbs/arch.h) some
>>> providers disabled!")"
>>
>> Yep, it's there, lost it in all the verbosity. Okay, that explains that.
>
> FWIW, that section after 'Missing Optional Items:' reflects everything
> 'wrong' with the build environment. An ideal build will list nothing,
> eg on FC25.
>
> Everything else means something is broken and cmake is working around
> the breakage somehow. It is worth reviewing if the thing is really
> broken or not...

 From the s390x build log:

-- Missing Optional Items:
--  Architecture NOT able to do coherent DMA (check libibverbs/arch.h) 
some providers disabled!
--  C11 stdatomic.h NOT available (old compiler)
--  rdma/vmw_pvrdma-abi.h NOT found (old system kernel headers)
--  -Wmissing-field-initializers does NOT work

The old compiler part isn't going to change, and we don't have the 
vmware pvrdma driver backported at this time.

-- 
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html